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FOREWORD 


The  use  of  simulations  in  U.S.  Army  training  continues  to  increase,  as  does  the  need  for 
tools  and  techniques  for  exploiting  simulation  training  capabilities.  For  the  past  several  years  the 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  has  been  a  leader  in  the 
development  of  structured  training  approaches  providing  such  tools  and  techniques,  primarily 
through  work  accomplished  in  the  Armored  Forces  Research  Unit  (AFRU)  at  Fort  Knox, 
Kentucky.  Experience  with  structured  simulation-based  training  has  led  to  the  recognition  of  a 
need  to  provide  a  comprehensive  system  to  “train  the  trainer.”  The  Close  Combat  Tactical 
Trainer  (CCTT)  magnifies  this  need  since  an  experienced  full-time  training  team  is  not  provided 
to  conduct  training.  Commanders  and  other  trainers  need  to  understand  the  capabilities  of  the 
CCTT,  and  to  be  able  to  tailor  structured  training  to  maximize  the  benefit  of  the  CCTT  in  their 
unit  training  strategy. 

This  report  describes  the  design,  prototype  development,  formative  evaluation,  and 
refinement  of  the  second  generation  of  a  computer  software  package  that  helps  commanders  and 
other  unit  trainers  to  develop  and  manage  structured  training  in  the  CCTT.  This  effort  was 
entitled  “Commanders’  Integrated  Training  Tool  -  2  (CITT-2),”  building  upon  an  earlier  AFRU 
project  which  led  to  development  of  the  initial  version  of  the  CITT.  The  AFRU  accomplished 
the  CITT-2  effort  as  part  of  Work  Package  205,  “Assessment  of  Force  XXI  Training  Tools  and 
Techniques.”  The  relevant  requirements  document  is  a  Memorandum  for  Record  between  the 
Chief,  AFRU  and  the  Project  Manager  for  the  Combined  Arms  Tactical  Trainer  (PM  CATT), 
entitled  “Structured  Training  for  the  Close  Combat  Tactical  Trainer,”  dated  25  July  1997. 

The  CITT-2  software  design  and  prototype  software  package  were  briefed  to 
representatives  of  the  U.S.  Army  Training  Support  Center,  the  PM  CATT,  and  the  Training  and 
Doctrine  Command  System  Manager  for  CATT  on  21  January  00.  The  second-generation 
software  package  is  available  in  standalone  and  Web-based  versions  and  has  been  provided  to 
CCTT  sites  and  other  selected  locations.  This  report  documents  the  methods  and  lessons  learned 
in  the  second  round  of  CITT  design,  and  in  developing  and  formatively  evaluating  the  second- 
generation  prototype.  It  will  be  useful  to  individuals  and  agencies  involved  in  the  development 
and  implementation  of  Army  training  development  and  management  systems  for  live,  virtual,  or 
constructive  training  environments. 


f  A  M.  SIMUTIS 
Technical  Director 
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THE  COMMANDERS’  INTEGRATED  TRAINING  TOOL  FOR  THE  CLOSE  COMBAT 
TACTICAL  TRAINER  -  2:  SECOND  GENERATION  DESIGN  AND  PROTOTYPE 
DEVELOPMENT 


EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  is  currently  fielding  the  Close  Combat  Tactical  Trainer  (CCTT)  as  the 
first  member  of  the  Combined  Arms  Tactical  Trainer  family.  The  CCTT  provides  a  virtual 
environment  supporting  the  collective  training  of  armored  and  mechanized  infantry  units.  To 
maximize  its  effectiveness,  the  CCTT  will  be  fielded  as  a  complete,  integrated  training  system, 
i.e.,  in  addition  to  the  basic  hardware  and  software  that  comprise  the  system,  it  will  also  provide 
the  tools  required  to  enable  its  users  to  achieve  maximum  benefit  from  its  use.  As  CCTT 
training  tools,  techniques,  and  procedures  have  evolved,  the  need  has  increased  for  integrating 
them  so  that  commanders  and  other  unit  trainers  can  access  and  use  them  readily  and  effectively. 
Such  an  integrating  system  or  tool  should:  (a)  provide  trainers  with  ready  access  to  all  the 
information  and  methods  they  need  to  exploit  the  emerging  capabilities  of  CCTT;  (b)  be 
compatible  with  Army  training  management  information  systems  and  databases;  (c)  lead  users  to 
effective  and  efficient  methods  for  developing  and  implementing  training  by  providing  ready 
access  to  available  exercises,  associated  Training  Support  Packages  (TSPs)  and  other  materials; 
(d)  provide  users  with  an  understanding  of  and  means  to  apply  a  structured  approach  to  meeting 
training  requirements;  and  (e)  address  the  training  of  digital  forces. 

From  October,  1997  to  January,  1999  a  project  to  research  and  develop  the  Commanders 
Integrated  Training  Tool  (CITT)  for  the  CCTT,  a  tool  having  the  characteristics  described  above, 
was  completed.  This  project  resulted  in  the  design  and  development  of  the  CITT.  The  design 
was  presented  in  the  form  of  Integrated  Definition  (IDEF)  models  and  Node  Tree  Diagrams;  the 
prototype  was  delivered  as  a  desktop  software  application,  a  Web  Site,  and  two  information 
videos.  As  the  project  was  being  completed,  several  new  research  and  development  (R&D) 
requirements  were  identified  including  the  need  to  address  more  than  one  terrain  database 
(TDB),  the  need  to  include  requirements  for  Force  XXI  Battle  Command  Brigade  and  Below 
(FBCB2)  equipped  units,  and  the  need  to  integrate  the  CCTT  Exercise  Initialization  Tool  (CEIT) 
into  CITT.  These  were  addressed  in  the  present  project. 

Procedure: 

The  project  objectives  were  accomplished  through  the  completion  of  six  major  activities. 
During  the  first  month  of  the  project,  a  comprehensive  R&D  plan  was  produced  and  provided  to 
the  project  Contracting  Officer’s  Representative  for  approval.  Upon  approval,  development  of 
the  expanded  CITT  design  was  initiated.  Design  activities  were  accomplished  using  U.S.  Army 
approved  methodologies,  such  as  IDEF  modeling.  Concurrently  with  CITT  design,  a  second- 
generation  prototype  CITT  was  developed.  Development  occurred  in  two  phases-development 
and  fielding  of  a  refined  CITT  to  support  formative  evaluation  (FE)  and  development  of  the 
second-generation  prototype  CITT  based  on  requirements  stated  in  the  R&D  plan  and  on  the 
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results  of  FE.  Development  occurred  using  the  same  methodologies  as  had  been  employed  in  the 
original  CITT  project.  The  FE  was  accomplished  through  a  combination  of  internal  testing,  and 
assessment  of  CITT  using  personnel  from  several  Active  Component  units.  Refinement  and 
expansion  of  the  implementation  strategy  and  fielding  plan  for  the  CITT  was  accomplished  by 
developing  near-,  mid-,  and  long-term  plans  which  take  into  account  current  and  anticipated 
Army  requirements  and  changes  in  technology.  The  final  activities  of  the  project  involved 
documentation  of  project  activities,  lessons  learned,  and  recommendations  relating  to 
implementation  and  fielding. 

Findings: 

The  CITT  design  was  completed  and  documented  from  the  standpoint  of  the  unit  trainer 
as  the  CITT  “To-Be”  design  and  is  documented  as  a  separate  U.S.  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences  (ARI)  Research  Note  (Dannemiller  &  Gossman,  in 
preparation).  The  design  addresses  a  “future”  CITT  software  application  modeled  from  the 
viewpoint  of  the  unit  commander/trainer.  A  prototype  CITT  desktop  application  and  a  CITT 
Web  Site  were  also  developed.  The  prototype  CITT  2.0  was  fielded  at  a  limited  number  of 
locations  in  April  2000.  The  CITT  Web  Site  is  currently  operational.  The  design,  prototype 
CITT,  and  the  Web  Site  address  the  expansion  of  CITT  as  described  in  the  CITT  Statement  of 
Work  (ARI,  1999)  including  additional  TDBs,  FBCB2-equipped  units,  and  the  CEIT/CrrT 
interface.  Implementation  strategies  and  fielding  plans  for  near-,  mid-,  and  long-term  were 
developed,  and  lessons  learned  were  documented. 

Utilization  of  Findings: 

The  specific  audiences  who  will  find  the  information  contained  in  this  report  beneficial 
include:  (a)  designers  and  developers  who  continue  further  development  of  the  CITT,  (b) 
training  unit  and  CCTT  training  site  personnel,  (c)  simulation  system  developers,  and  (d)  any 
member  of  the  U.S.  Army  who  wants  to  better  understand  the  TSP  development  process.  The 
CITT  design  is  fully  documented  and  can  be  used  as  the  basis  for  the  development  of  an 
integrated  training  tool  under  any  of  several  fielding  alternatives  developed  as  part  of  the  project. 
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THE  COMMANDERS’  INTEGRATED  TRAINING  TOOL  FOR  THE  CLOSE 

COMBAT  TACTICAL  TRAINER  -  2:  SECOND  GENERATION  DESIGN  AND 

PROTOTYPE  DEVELOPMENT 

Introduction 

Beginning  in  October  1997,  the  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences  (ARI)1  Armored  Forces  Research  Unit  (AFRU)  at  Fort  Knox,  Kentucky, 
sponsored  a  project  to  develop  the  Commanders’  Integrated  Training  Tool  (CITT)  for  the  Close 
Combat  Tactical  Trainer  (CCTT).  The  CCTT  is  a  virtual  training  system  that  supports  the 
training  of  collective  tasks  for  armored  and  mechanized  infantry  units,  including  combat  support 
and  combat  service  support,  at  the  platoon  and  company/team  level.  It  includes  the  capability  to 
support  battalion  task  force  (and  perhaps  brigade)  training  as  command  field  exercises  or  as 
portions  of  larger  integrated  exercises.  For  training  in  CCTT  to  be  maximally  effective,  it  must 
incorporate  the  “Train  as  You  Fight”  doctrine  of  FM  25-100  (Department  of  the  Army,  1988) 
and  FM-25-101  (Department  of  the  Army,  1990)  and  follow  the  principles  of  structured  training 
that  have  been  developed  over  the  past  several  years  (Campbell,  Campbell,  Sanders,  Flynn  & 
Myers,  1995;  Bessemer  &  Myers,  1998;  Wilkinson,  in  preparation).  Indeed,  over  60  structured 
training  exercises  have  been  developed  by  contractor  teams  working  primarily  at  Fort  Knox 
under  the  sponsorship  of  ARI  (Flynn,  Campbell,  Myers,  and  Burnside,  1998,  Deatz  et  al.,  1998). 
These  exercises  and  their  related  training  support  packages  (TSPs)  provide  the  core  set  of 
exercises  around  which  CCTT  training  and  testing  have  been  conducted. 

The  CITT  project  (Gossman  et  al.,  1999)  provided  a  detailed  design  and  prototype  for  a 
tool  that  allows  commanders  and  other  unit  trainers  to  tailor  the  training  their  unit  will  conduct  in 
CCTT  to  their  specific  unit  needs.  This  is  accomplished  by  allowing  commanders  and  other 
trainers  to  select  from  existing  exercises  if  such  exercises  match  their  training  needs;  by 
modifying  an  existing  exercise  to  more  closely  match  their  training  needs;  or  by  creating  an 
entirely  new  exercise  that  matches  their  training  needs.  In  addition,  the  CITT  provides  extensive 
information  on  the  CCTT  and  on  developing  structured  training  exercises.  It  is,  in  effect,  a 
comprehensive  source  of  information  needed  by  users  to  participate  effectively  and  efficiently  in 
CCTT  training.  It  is  also  a  TSP  authoring  tool. 

At  the  conclusion  of  the  CITT  project,  a  number  of  recommendations  for  improvement  to 
the  CITT,  both  the  design  and  prototype,  were  included  in  the  final  report.  For  example, 
formative  evaluation  (FE)  data  indicated  that  the  TSP  authoring  process  was  overly  complex,  and 
the  graphical  user  interface  (GUI)  employed  in  the  prototype  was  somewhat  cumbersome.  In 
addition,  a  number  of  new  requirements  for  the  CITT  had  been  identified  by  ARI  in  the  CITT-2 
Statement  of  Work  ([SOW]  ARI,  1999).  Specifically,  there  was  a  need  to  incorporate  support 
for  the  CCTT  capability  to  train  Force  XXI  Battle  Command  Brigade  and  Below  (FBCB2) 
equipped  units;  there  was  a  need  to  add  the  Temperate  Forest  (Primary  1  [PI])  and  Fort  Hood 
(P3)  TDBs  to  the  CITT;  and  there  was  a  need  to  examine  and  design  effective  and  efficient 
means  for  an  interface  between  CITT  users  and  CCTT  site  personnel. 


1  A  list  of  all  acronyms  used  in  the  report  is  included  in  Appendix  A. 
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In  January  1999,  ARI AFRU  initiated  a  project  to  expand  and  enhance  the  CITT  design 
and  to  produce  a  second-generation  prototype  CITT  which  would  incorporate  the 
recommendations  from  the  initial  CITT  project  as  well  as  the  additional  requirements  specified 
above.  This  project  was  designated  the  Commanders’  Integrated  Training  Tool  for  the  Close 
Combat  Tactical  Trainer  -  2  (CITT-2)  and  is  the  subject  of  this  research  report. 

Purpose  of  the  Report 

The  purpose  of  this  report  is  to  describe  the  research  methods  and  outcomes  of  the  project 
to  produce  the  design  for  an  expanded  CITT  and  to  develop  a  second-generation  CITT  prototype 
in  stand-alone  and  distributed  forms.  Additional  project  activities  and  objectives  will  be 
described,  such  as  enhancements  to  the  CCTT  Instructional  Overview  (IO),  the  incorporation  of 
expanded  help  and  tutorial  files  into  the  CITT,  the  conduct  of  FE  in  support  of  design  and 
prototype  development,  and  lessons  learned  that  can  be  applied  to  similar  projects. 

Organization  of  the  Report 

This  report  is  organized  as  follows: 

1.  The  Project  Background  and  Need  section  describes  the  need  for  the  expanded  and 
enhanced  CITT  and  the  overall  purpose  of  the  project  including  a  statement  of  the  project 
objectives. 

2.  The  Research  Methodology  section  describes  the  activities  that  were  completed  and 
the  methods  that  were  employed  to  achieve  the  project  objectives.  The  first  section  focuses  on 
the  design  of  the  enhanced  CITT;  this  is  followed  by  a  description  of  the  development  of  the 
second-generation  CITT  prototype  (Versions  1.1  and  2.0)  including  the  expanded  IO  and  help 
functions;  the  next  section  focuses  on  the  FE  conducted  to  support  prototype  development;  and 
the  final  section  describes  the  activities  involved  in  development  of  the  near-,  mid-,  and  long¬ 
term  implementation  strategy  and  fielding  plans. 

3.  The  Results  section  focuses  on  the  products  and  findings  of  the  project.  Its 
organization  mirrors  the  Research  Methodology  section  and  describes  the  outcomes  of  the 
design,  development,  and  FE  activities.  It  also  describes  the  implementation  strategy  and 
fielding  plan  for  the  CITT. 

4.  The  Lessons  Learned  section  describes  findings  from  the  project  that  are  relevant  to 
future  design  and  development  efforts. 

5.  The  Summary  and  Recommendations  section  provides  a  brief  synopsis  of  the  report 
and  presents  a  number  of  recommendations  for  future  efforts. 

Project  Background  and  Need 

With  the  successful  completion  of  the  CITT  project,  there  was  a  need  to  update  and 
expand  the  CITT  design  and  the  CITT  prototype  both  as  a  result  of  findings  from  the  initial 
CITT  project  and  as  a  result  of  new  developments  in  CCTT  and  related  technology.  A  list  of  33 
findings  and  recommendations  for  improvements  to  the  CITT  was  included  in  the  CITT  final 
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report  (Gossman  et  al.,  1999)  in  addition  to  the  general  findings  cited  previously  that  the  user 
interface  was  cumbersome  and  the  TSP  development  process  was  somewhat  confusing.  There 
was  a  need  to  address  these. 

External  to  the  CITT  project  and  findings,  a  number  of  additional  developments  had 
occurred  or  were  occurring  that  further  emphasized  the  need  to  expand  and  enhance  the  CITT. 
For  example,  the  initial  development  of  training  methods  and  tools  for  the  CCTT  (including 
CITT)  focused  on  the  training  of  conventional  forces.  Recently,  capabilities  for  training  digital2 
(FBCB2)  units  have  been  incorporated  into  the  CCTT  through  the  CCTT  XXI  initiative.  The 
CITT  design  and  prototype  required  expansion  to  address  these  capabilities  fully,  drawing  from 
relevant  initiatives  such  as  the  ARI ARFU  effort  entitled  “Training  for  the  Digital  Battlefield” 
(Dierksmeier  et  al.,  1999)  and  an  ongoing  effort  to  develop  FBCB2  TSPs  for  CCTT  under  the 
Force  XXI  Training  Program  at  Fort  Knox.  As  another  example  of  needed  CITT  expansion, 
initial  CCTT  TSPs,  CITT  design,  and  the  CITT  prototype  focused  on  the  National  Training 
Center  (P2)  TDB  initially  available  in  CCTT.  The  Temperate  Forest  (PI)  and  Fort  Hood  (P3) 
TDBs  were  becoming  available,  and  CITT  needed  to  be  expanded  to  support  development  of 
TSPs  for  exercises  on  them. 

Based  on  these  needs,  ARI  issued  a  SOW  for  the  Commanders’  Integrated  Training  Tool 
for  the  Close  Combat  Tactical  Trainer  -  2  (CITT-2)  project.  The  stated  purpose  of  the  project 
was  to  update  and  expand  CITT  to  support  commanders  and  other  unit  trainers  in  exploiting  fully 
the  capabilities  of  CCTT  for  training  conventional  and  digital  forces.  The  CITT-2  project  was  to 
sustain  the  initial  CITT  focus  at  the  company  team  and  platoon  levels  as  well  as  continuing 
design  and  development  within  a  brigade  training  context  addressing  the  fit  of  CCTT  in  a  global 
training  strategy  in  order  to  provide  an  appropriate  tactical  training  framework  and  to  support 
possible  further  expansion.  The  project  was  to  result  in  an  expanded  CITT  describing  fully  and 
facilitating  the  implementation  of: 

1.  The  current  and  emerging  capabilities  of  CCTT  to  train  conventional  and  FBCB2- 
equipped  units; 

2.  The  methods  (including  cognitive  aspects-the  “why”  as  well  as  the  “how”)  of 
applying  the  structured  training  principles  and  approach  to  developing  and  implementing  training 
in  CCTT;  and 

3.  The  tools,  techniques,  and  procedures  for  training  efficiently  and  effectively  with 

CCTT. 


This  overall  purpose  was  to  be  accomplished  through  the  achievement  of  five  project 
objectives: 

1.  To  update  and  expand  the  initial  design  for  the  CITT  in  stand-alone  and  distributed 
versions  including  digital  (FBCB2)  training  capabilities  of  the  CCTT,  the  inclusion  of  multiple 
TDBs,  and  the  CITT  user-CCTT  site  interface. 


2  Note:  for  convenience  the  terms  “digital  unit”  and  “FBCB2-equipped  unit”  are  used  interchangeably  in 
this  report. 
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2.  To  develop  a  second-generation  prototype  CITT  in  stand-alone  and  distributed 
versions  incorporating  the  updated  and  expanded  design  to  the  extent  possible. 

3.  To  refine  the  second-generation  prototype  CITT  and  design  documentation,  based  on 
the  conduct  of  FE,  by  providing  support  for  and  monitoring  limited  implementation  focusing  on 
the  stand-alone  version. 

4.  To  refine  and  expand  the  implementation  strategy  and  fielding  plan/methods  for  the 

CITT. 

5.  To  document  results  of  second-generation  CITT  design  and  development  including 
lessons  learned  and  recommendations  relating  to  further  implementation  and  expansion  of  the 
CITT. 


The  CITT  Team  produced  a  response  to  the  SOW  in  the  form  of  a  Research  Program 
Plan  (CITT  Team,  1999d),  which  described,  in  detail,  the  methodology  for  accomplishing  these 
objectives. 


Research  Methodology 

This  section  focuses  on  the  process  and  activities  through  which  the  project  goals  and 
objectives  were  achieved  including  describing  the  research  methods  employed  to  obtain  and 
analyze  data  as  well  as  the  key  decisions  that  were  made  based  on  the  data  collected  and 
analyzed.  Figure  1  shows  the  project  schedule  for  completion  of  the  activities  described  in  this 
section. 


Figure  1.  CITT-2  project  timeline. 
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Design  the  Enhanced  CITT 


Using  lessons  learned  from  the  original  CITT  research  and  development  (R&D)  project 
as  well  as  technological  advances  that  had  occurred  since  the  development  of  the  CITT  Stand¬ 
alone  1.0,  the  CITT  Team  expanded  the  design  of  the  CITT  for  the  CCTT.  Initial  design 
enhancements  were  intended  to  represent  the  CITT  1.1  prototype  (see  below).  This  included  the 
addition  of  information,  data,  and  forms  required  for  FBCB2-equipped  units,  multiple  TDBs,  the 
CITT  user-CCTT  site  interface,  and  the  expanded  IO.  The  methodology  employed  was 
essentially  the  same  as  that  used  in  the  CITT  project.  Members  of  the  team  collected  and 
analyzed  information  regarding  proposed  enhancements  including  how  the  enhancements  would 
be  incorporated  into  the  CITT  design.  The  team  also  worked  closely  with  a  contractor  team 
which  was  working  on  a  Program  Manager-Combined  Arms  Tactical  Trainer  (PM-CATT) 
funded  project  to  develop  the  CCTT  Exercise  Initialization  Tool  (CEIT).  During  the  initial  CITT 
project,  the  team  had  worked  with  this  contractor  group  which,  at  that  time,  was  developing  a 
map  and  overlay  tool  for  use  in  developing  CCTT  exercises.  This  tool  was  incorporated  into 
CITT  1.0.  As  a  follow-on  to  the  map  and  overlay  tool  project,  contractors  from  Lockheed- 
Martin  and  AcuSoft  (hereafter  called  the  CEIT  development  team)  were  tasked  with  developing 
CEIT  which  includes  enhanced  map  and  overlay  tools  as  well  as  the  ability  to  generate  a  CCTT 
exercise  initialization  file  from  the  maps  and  overlays  produced.  This  initialization  file  provides 
a  significant  component  of  the  CITT  user-CCTT  site  interface. 

Using  Department  of  Defense  and  Department  of  the  Army  (DA)  approved  methodology 
for  Integrated  Definition  (EDEF)  modeling  (DA,  1997;  U.S.  Department  of  Commerce,  1993), 
the  design  was  completed  and  documented  using  the  following: 

1.  IDEFO-used  to  produce  a  function  or  activity  model  that  is  a  structured  representation 
of  the  expanded  CITT  functions. 

2.  IDEFIX-used  to  document  the  modifications  to  the  relational  database  schema  of  the 
expanded  CITT.  The  IDEF1X  models  are  applicable  only  for  the  initial  design. 

3.  IDEF3  and  Data  Flow  Diagram-used  to  capture  descriptions  of  activity  sequences  and 
details  of  the  function  or  activity  model. 

The  initial  design  was  briefed  by  the  CITT  team  during  an  in-process  review  conducted 
in  May  1999  and  was  approved  by  the  Contracting  Officer’s  Representative  (COR)  and  other 
interested  parties. 

Development  of  the  expanded  CITT  design  continued  for  the  remainder  of  the  project 
with  much  of  the  work  centering  around  the  inclusion  of  the  CEIT  into  the  design.  The  CITT 
team  worked  closely  with  the  CEIT  development  team  to  identify  and  define  how  CEIT  and 
CITT  interact  and  how  CEIT  could  be  incorporated  into  the  expanded  CITT.  Additionally, 
enhanced  design  included  deriving  user  and  system  requirements,  identification  of  key  issues  and 
functions/activities  of  the  expanded  CITT,  determining  new  and/or  additional  interfaces,  and 
allocation  of  architecture  requirements  to  system  elements. 
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Develop  Second-Generation  CITT  Prototype 


As  shown  in  Figure  1,  the  second-generation  CITT  Prototype  development  consisted  of 
two  separate  phases:  development  of  an  enhanced  second-generation  CITT  for  limited  fielding 
to  support  FE;  and  development  of  a  refined  second-generation  CITT,  based  on  the  results  of  FE, 
for  implementation  and  fielding  at  the  conclusion  of  the  project.  (For  the  remainder  of  this 
report,  the  enhanced  second-generation  CITT  will  be  referred  to  as  the  CITT  1.1,  and  the  refined 
second-generation  CITT  will  be  referred  to  as  CITT  2.0.) 

CITT  1.1 


The  CITT-2  SOW  specified  that,  at  a  minimum,  the  CITT  1.1  would  provide 
commanders  and  other  unit  trainers  with  the  capabilities  to  access,  modify,  or  develop  CCTT 
exercises  on  the  PI,  P2,  and  P3  TDBs  for  conventional  and  FBCB2-equipped  platoons  and 
company  teams.  The  CITT  1.0  (the  final  product  of  the  CITT  project)  had  addressed  only  the  P2 
TDB  and  did  not  address  FBCB2  at  all. 

To  address  FBCB2,  the  team  was  required  to  collect  and  analyze  data  on  its  operation  and 
capabilities  in  CCTT  and,  from  this,  to  determine  what  was  appropriate  to  include  in  CITT.  Data 
were  collected  from  three  primary  sources.  First,  the  team  obtained  and  reviewed  FBCB2- 
related  documentation  (TRW  Systems  and  Information  Technology  Group,  1998;  DA,  1998). 
Second,  a  copy  of  the  FBCB2  software  (Version  2.0  Beta)  was  installed  on  a  local  computer 
allowing  members  of  the  team  to  analyze  its  operation  including  message  flow,  data  transfer,  etc. 
Third,  two  members  of  the  team  observed  the  FBCB2  limited  user  test  at  the  Fort  Hood  CCTT 
site  which  allowed  them  to  obtain  data  on  how  units  actually  used  the  system  while  running 
CCTT  exercises. 

Data  collected  from  the  various  sources  were  synthesized  to  determine  their  impact  on 
the  CCTT  TSP  content,  structure,  and  format  developed  during  the  Structured  Training  for  Units 
in  the  Close  Combat  Tactical  Trainer  projects  (Flynn,  Campbell,  Myers,  and  Burnside,  1998; 
Deatz  et  al.,  1998).  In  addition,  one  member  of  the  team  was  temporarily  assigned  to  assist  in 
the  development  of  two  structured  training  TSPs  for  FBCB2-equipped  units,  and  the  information 
from  this  experience  facilitated  the  process  of  determining  the  data  needed  in  the  TSP.  Once 
TSP  additions/modifications  were  identified,  an  analysis  was  completed  to  determine  which 
should  be  supported  by  or  included  in  the  CITT  1.1. 

The  inclusion  of  alternate  TDBs  was  approached  from  the  standpoint  of  adding  the  maps 
that  were  available  at  the  time  in  order  to  support  limited  fielding  of  CITT  1.1  in  early  June 
1999.  Six  scanned  images  depicting  the  lower  right  section  of  the  PI  TDB  were  obtained  from 
U.S.  Army  Simulation,  Training,  and  Information  Command  (STRICOM)  and  were  combined  to 
form  a  single  graphic  image  which  was  then  incorporated  into  the  CITT  application.  In  order  to 
use  this  map,  a  function  to  select  the  PI  or  P2  TDB  was  added  to  the  PowerPoint  map  and 
overlay  tools  that  had  been  included  in  CITT  1.0.  In  addition,  the  source  code  for  the  map  and 
overlay  tools  was  modified  to  eliminate  a  problem  with  scaling  of  graphic  images  when  the  user 
changed  TDBs.  The  programming  necessary  to  accomplish  both  was  completed  prior  to  fielding 
of  the  CITT  1.1.  Asa  test  of  the  operation  of  the  PI  map  and  its  incorporation  into  CITT  1.1, 
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two  CCTT  exercise  TSPs  were  developed  by  converting  exercises  based  on  the  P2  TDB  to  the 
PI  TDB. 

No  attempt  was  made  to  incorporate  the  P3  TDB  or  the  other  sections  of  the  P2  TDB  into 
CITT  1.1.  This  decision  was  made  based  on  information  from  the  CEIT  development  team  that 
the  PM-CATT  was  in  the  process  of  converting  all  terrain  maps  into  digitized  plan  view  displays 
(PVDs)  consistent  with  those  used  in  CCTT.  The  team  decided  to  delay  adding  additional  terrain 
maps  until  the  PVDs  were  available. 

Following  completion  of  data  collection  and  analysis,  the  CITT  application  was 
modified.  The  changes  to  the  TSP  necessary  to  support  CCTT  training  of  digital  units  were 
incorporated  by  revising  existing  tables,  forms,  and  queries;  five  Mission  Training  Plans  (MTPs) 
needed  to  support  training  of  digital  units  were  added  to  the  existing  database;  Help  files  were 
updated  to  reflect  the  modification  made  to  the  CITT;  and  the  10  was  updated  to  include  FBCB2 
and  PI  TDB  information.  Throughout  the  modification  process,  the  CITT  was  tested  by 
members  of  the  development  team.  This  testing  consisted  of  exercising  the  application  to 
determine  if  it  would  “break,”  i.e.,  if  it  would  produce  errors  in  the  Access  database  or  in  the 
TSP  authoring  process.  All  errors  discovered  were  entered  into  a  Census  97  database  which  had 
been  set  up  to  track  errors. 

Two  short  booklets  were  produced  (CITT  Team,  1999b,  1999c).  These  contained  basic 
information  to  assist  CITT  administrators  and  users  in  installing  CITT  should  that  be  necessary 
and  in  using  CITT,  particularly  users  unfamiliar  with  using  Microsoft  Access-based  applications. 
At  this  point,  the  CITT  1.1  underwent  limited  fielding  to  support  FE  as  described  below. 

CITT  2.0 


Work  on  CITT  2.0  began  with  the  identification  of  several  goals  that  the  team  wanted  to 
achieve.  First,  the  enhanced  CITT  2.0  would  incorporate  findings  from  the  FE  of  previous  CITT 
systems  1.0  and  1.1.  Second,  CITT  2.0  would  include  a  simplified  user  interface  and  a 
simplified  TSP  development/authoring  process.  And  finally,  it  would  incorporate  more  military 
terminology  rather  than  the  simulation  and  CCTT  terminology  that  had  been  used  in  earlier 
versions.  For  the  remainder  of  2.0  development,  these  goals  became  constraints  in  that  proposed 
modifications  and  additions  to  CITT  were  continuously  evaluated  in  terms  of  meeting  them. 

Two  primary  data  sources  were  initially  examined  to  identify  requirements  for  the  CITT 
2.0.  The  final  report  for  CITT  had  identified  33  recommendations  for  improvement  to  CITT,  and 
internal  testing  of  CITT  1.1  had  identified  15  defects  and/or  modifications  that  had  been  deferred 
to  CITT  2.0.  These  48  modifications/improvements  were  analyzed  and  classified  into  two 
categories:  those  that  could  be  incorporated  with  little  difficulty  and  those  that  could  be 
incorporated,  but  with  greater  difficulty.  They  were  prioritized  according  to  their  criticality  for 
inclusion  into  2.0.  Based  on  these  analyses,  a  list  of  development  tasks  for  producing  CITT  2.0 
was  developed,  and  a  timeline  for  accomplishing  them  was  produced.  These  were  presented  to 
the  COR,  and,  based  on  his  comments  and  recommendations,  a  final  list  and  timeline  were 
produced. 
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At  this  point  in  the  project,  the  requirement  to  coordinate  with  the  CEIT  project  was 
elaborated  further.  The  CEIT  is  a  major  extension  of  the  map  and  overlay  tools  that  had  been 
included  in  CITT  1.0.  In  addition  to  providing  the  mapping  tools  in  an  enhanced  version,  CEIT 
is  intended  to  “interpret”  information  on  the  map  (e.g.,  unit  locations,  unit  types,  opposing  forces 
[OPFOR]  and  blue  forces  units  and  locations,  environmental  data,  etc.)  and  produce  an  American 
Standard  Code  for  Information  Interchange  file  which  can  be  used  to  initialize  an  exercise  at  a 
CCTT  site.  The  CEIT  provides  a  valuable  link  between  the  unit  trainer  who  is  developing  an 
exercise  using  CITT  and  the  CCTT  site  staff  who  will  manage  and  run  the  exercise  at  the  site. 

The  CEIT  had  several  significant  impacts  on  2.0  development.  Its  most  immediate 
impact  was  the  additional  time  required  by  the  team  to  gather  and  analyze  data  on  CEIT  and  how 
it  operates.  This  was  necessary  to  make  timely  decisions  on  how  best  to  approach  incorporation 
of  CEIT  and  how  to  coordinate  with  the  CEIT  development  team.  Once  initial  decisions  were 
made,  incorporation  of  CEIT  became  an  ongoing  requirement  of  the  project  similar  to  other 
identified  requirements.  The  major  difference  between  CEIT  and  other  project  requirements  was 
the  much  higher  level  of  coordination  required  between  the  two  projects  due  to  the  fact  that  both 
projects  were  ongoing  and  there  was  a  high  degree  of  interdependence  between  them. 

The  CEIT/CITT  coordination  occurred  continuously  from  the  time  the  requirement  to 
integrate  CEIT  into  CITT  was  specified.  Information  was  shared  on  an  almost  daily  basis  via 
e-mail  and  telephone  communications.  In  addition,  four  site  visits  occurred:  two  occasions  on 
which  CITT  team  members  visited  the  CEIT  team  in  Orlando;  and  two  occasions  on  with  CEIT 
team  members  visited  the  CITT  team  at  Fort  Knox.  These  meetings  occurred  in  April,  July,  and 
August  1999,  and  January  2000.  In  the  interim,  numerous  conference  calls  were  held  to  ensure 
ongoing  coordination.  This  level  of  coordination  was  effective  in  achieving  the  degree  of 
integration  of  CEIT  into  CITT  that  occurred;  however,  integration  was  always  limited  by  the  fact 
that  the  end  date  for  CEIT  was  three  months  after  the  end  date  for  CITT-2. 

CEIT  integration  required  the  identification  of  common  data  elements,  i.e.,  data  included 
in  both  the  CEIT  and  CITT  databases;  specifically  the  data  that  CEIT  takes  from  the  maps  and 
overlays  produced  using  the  CEIT  embedded  in  CITT  and  converts  to  the  exercise  initialization 
file  that  is  loaded  into  CCTT.  Close  coordination  allowed  the  two  teams  to  identify  elements 
used  by  both  CEIT  and  CITT  and  to  devise  a  process  for  making  the  data  available  to  both 
applications.  The  process  involved  the  creation  of  a  “shared  database.”  The  shared  database 
consists  of  such  data  elements  as  unit  identification,  starting  locations,  environment,  and  unit 
frequencies  and  radio  nets,  and  is  essentially  a  buffer  that  stores  these  data  elements  as  they  are 
generated  by  CEIT  and  can  then  be  “read”  by  CITT  and  used  to  populate  the  appropriate  CITT 
tables.  In  so  doing,  these  data  become  part  of  the  TSP  without  requiring  additional  input  on  the 
part  of  the  CITT  user. 

Once  the  analysis  of  CITT  2.0  requirements,  including  CEIT,  was  substantially  complete, 
work  on  prototype  development  of  2.0  was  initiated.  “Mock-up”  screen  shots  were  developed 
for  the  complete  CITT  process.  After  an  internal  review  and  modification,  revised  screen  shots 
were  developed  and  presented  to  a  focus  group  comprised  of  representatives  from  the  military, 
CCTT  site  personnel,  and  other  stakeholder  organizations.  This  focus  group  was  similar  to  the 
user  jury  that  had  been  used  in  the  original  CITT  project;  however,  based  on  findings  from  the 
CITT  project,  the  review  process  was  conducted  somewhat  differently.  First,  there  were  fewer 
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meetings  of  the  group;  however,  meetings  were  substantially  longer  (three  to  four  hours  versus 
one  hour)  and  more  focused.  The  meetings  had  fixed  agendas,  and  the  group  facilitator  generally 
kept  the  group  focused.  This  allowed  the  group  adequate  time  to  consider  and  discuss  the 
information  presented.  Second,  the  CITT  team  provided  the  group  with  specific  input  on  both 
the  TSP  development  process  and  the  proposed  GUI  which  further  helped  focus  discussion.  The 
outcome  of  the  focus  group  meetings  was  a  list  of  proposed  modifications  to  the  CITT  (see 
below  for  more  detail),  including  renaming  graphics,  adding  task  conditions  and  standards, 
adding  a  “Why”  button  feature  to  address  the  cognitive  aspects  of  structured  training 
development,  and  clarification  of  the  functional  hierarchy  of  mission/exercise/tasks. 

Based  on  input  from  the  focus  group  and  continuing  data  collection  and  analysis 
including  data  from  the  PM-CATT,  final  mock-ups  of  screen  shots  portraying  the  TSP 
development  process  and  GUI  were  produced  and  a  timeline  and  process  for  2.0  prototype 
development  was  established.  This  process  consisted  of  the  following  steps: 

1.  Modify  the  underlying  CITT  database  tables  and  structure. 

2.  Build  screens  and  queries  using  Microsoft  Access. 

3.  Modify/develop  Help  screens  and  the  IO  and  develop  “Why?”  screens  and  tutorials. 

4.  Integrate  CEIT. 

5.  Re-enter  all  TSPs  in  CITT  1.1  in  order  to  produce  the  exercise  initialization  file. 

6.  Conduct  internal  testing. 

7.  Field  CITT  2.0. 

Development  of  CITT  2.0  generally  followed  this  process,  although,  as  would  be 
anticipated,  there  was  some  overlap  among  the  activities.  In  addition,  it  was  possible  to  segment 
at  least  part  of  the  development  along  functional  lines.  That  is,  once  modifications  to  the  basic 
database  were  complete,  the  remaining  activities  could  be  completed  for  the  “Create”  function, 
then  for  the  “Modify”  function,  etc.  The  need  for  “Why?”  screens  and  the  tutorials  referred  to  in 
step  3  had  come  out  of  the  focus  group  discussions  as  well  as  FE  of  earlier  versions.  The  team 
completed  the  data  collection  and  analysis  necessary  to  determine  how  these  should  be  used, 
how  they  should  function,  and  how  best  to  develop  them. 

As  development  proceeded,  several  decisions  were  made  which  impacted  the  final 
product.  The  Team  decided  to  remove  Education  of  CCTT  through  Computer  Assisted  Training 
Technology,  a  set  of  instructional  programs  for  training  CCTT  workstation  operators,  from  CITT 
since  the  long-term  plan  for  CCTT  is  to  combine  workstations  and  have  them  operated  by  site 
personnel.  The  browser  was  upgraded  to  Microsoft  Internet  Explorer  5  (IE5)  and  Microsoft 
Agent  was  upgraded  to  Version  2.  The  Navigation  function  was  restructured  to  provide  better 
information  in  a  more  timely  manner  to  CITT  users.  The  Build  and  Proof  function  was  removed 
from  the  CITT,  and  CCTT  Site  Tools  was  included  in  its  place.  In  conjunction  with  this 
decision,  the  team  completed  a  data  collection  effort  at  the  CCTT  sites  at  Forts  Benning,  Hood, 
and  Knox.  This  effort  was  directed  at  obtaining  data  from  appropriate  site  personnel  regarding 
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those  exercise  management  tools  that  would  be  of  greatest  value  to  them.  It  is  anticipated  that 
the  major  modifications  to  Navigate  and  CCTT  Site  Tools  will  appear  in  future  developments  of 

cm. 


The  need  for  one  additional  product  of  CITT  2.0  development,  not  included  in  the 
Research  Program  Plan,  became  apparent  as  development  proceeded.  Based  on  feedback  from 
Army  personnel  obtained  during  FE  and  from  additional  discussions,  the  team  decided  that  it 
would  be  of  benefit  to  potential  CITT  users  if  there  was  a  means  to  disseminate,  independently 
of  Cm,  the  IO  and  the  two  videos  (U.S.  Army,  1999a,  1999b)  produced  during  the  CITT 
project.  Digitized  versions  of  the  two  videos  were  obtained  and  converted  to  a  streaming  video 
format.  Several  iterations  of  conversion  were  completed  to  produce  a  file  small  enough  to  be 
used  efficiently  while  still  maintaining  a  high  level  of  quality  in  both  video  and  audio  output. 

The  final  conversion  resulted  in  files  of  less  than  100  Megabytes  reduced  from  original  files  that 
were  more  than  a  Gigabyte.  The  IO  was  modified  to  run  independently  of  CITT  and  directly 
from  a  CD-ROM,  and  a  CD  containing  the  IO  and  videos  was  produced  (CITT  Team,  1999a). 

Cm  Web  Site 

The  SOWs  for  CITT  and  CITT-2  specified  the  development  of  CITT  in  both  stand-alone 
and  distributed  versions,  and  in  the  original  CITT  project,  this  had  been  interpreted  as  developing 
essentially  the  same  system  that  users  could  run  either  on  a  local  machine  or  from  a  remote 
system  using  the  Internet.  Some  of  the  findings  from  the  CITT  project,  particularly  findings 
related  to  the  difficulty  of  accessing  the  Internet  from  an  Army  post,  caused  the  team  to  rethink 
the  whole  concept  of  the  distributed  CITT. 

The  team  analyzed  user  needs  for  information  and  training  in  light  of  the  best  use  of 
current  technology  and  concluded  that  the  development  of  TSPs  across  the  Internet  was  not  a 
particularly  good  use.  The  process  of  developing  TSPs  remotely  would  be  extremely  inefficient, 
primarily  because  it  would  be  a  very  slow  process  even  using  high-speed  links  to  the  Internet. 

The  process  would  involve  transfer  of  very  large  data  elements  and  would  probably  produce 
intolerable  waits  for  the  user. 

On  the  other  hand,  the  team  did  conclude  that  a  good  use  of  technology  would  be  to 
provide  a  site  from  which  users  could  obtain  the  information  they  need  to  make  decisions 
relative  to  their  training  needs.  The  team  also  decided  that  the  site  would  provide  an  ideal 
location  for  storing  and  accessing  existing  CCTT  exercises.  Based  on  these  decisions,  the  focus 
of  the  distributed  site  was  changed  to  being  an  information  center  and  TSP  repository,  and, 
consistent  with  this  change,  the  name  was  changed  to  the  CITT  Web  Site.  The  Web  Site  would 
fulfill  three  objectives:3 

1.  It  would  provide  all  of  the  information  about  CCTT,  structured  training,  CITT,  etc. 
needed  by  the  user  to  support  unit  training  needs. 


3  It  is  anticipated  that  when  the  CITT  is  no  longer  in  research  and  development,  these  objectives  will  be  met 
through  use  of  the  Reimer  Digital  Library. 
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2.  It  would  serve  as  a  repository  of  TSPs  approved  for  CCTT  training  and  would  provide 
a  mechanism  for  reviewing  and  selecting  TSPs  from  the  repository. 

3.  It  would  provide  a  file  transfer  capability  to  allow  users  to  download  TSPs  from  the 
site  and  to  upload  TSPs  to  the  site. 

Data  collection  and  analysis  to  determine  the  design  for  a  site  that  would  fulfill  these 
objectives  was  initiated.  Members  of  the  CITT  team  examined  existing  Internet  sites  to 
determine  features  that  would  function  well  on  the  Web  Site  as  well  as  a  number  of  references  on 
Internet  site  design  (Alden,  1998;  Davis  &  Merritt,  1998;  Driscoll,  1998;  Hall,  1997;  Morris  & 
Hinrichs,  1996;  Weinschenk,  Jamar,  &  Yeo,  1997).  They  further  decided  that  the  site  would  be 
fully  functional  from  either  IE5  or  Netscape  Navigator.  This  decision  placed  some  constraints 
on  the  development  of  the  site  since  there  are  features  in  IE5  that  are  not  presently  supported 
fully  in  Netscape.  The  team  examined  the  use  of  newsgroups  to  provide  a  means  for  users  to 
obtain  information  from  each  other.  They  looked  at  using  streaming  video  technology  for 
making  the  CITT  videos  accessible  via  the  site.  They  looked  at  various  navigation  methods 
including  the  use  of  cascading  menus.  And,  they  examined  various  formatting  and  graphics 
issues. 


Consistent  with  these  analyses  and  based  on  the  user  needs  identified,  the  team  began  the 
process  of  determining  the  structure  of  the  site.  They  determined  that  there  would  be  a  home 
page  from  which  the  user  would  access  the  CCTT  module,  the  CCTT  Videos,  TSPs,  and  the  file 
transfer  function.  The  CCTT  module  would  include  the  information  in  the  “Learn  About  CCTT” 
module  of  CITT;  the  CCTT  Videos  would  include  access  to  the  two  CITT  videos  in  their 
entirety;  TSPs  would  provide  access  to  the  complete  repository  of  CCTT  exercises;  and  the  file 
transfer  function  would  provide  the  mechanism  for  downloading  TSPs  from  the  site.  (The  team 
had  previously  decided  that  the  upload  function  would  be  provided  on  the  stand-alone  CITT.) 
Further  along  in  the  process  and  with  the  full  approval  and  input  of  CCTT  site  personnel,  the 
team  decided  that  an  individual  information  page  for  each  CCTT  site  would  be  provided  on  the 
Web  Site.  This  would  provide  a  central  location  where  users  could  obtain  information  they 
needed  to  support  their  CCTT  training  at  any  of  the  CCTT  sites.  Data  were  gathered  from  each 
site  to  provide  this  information. 

Based  on  the  data  collection  and  analysis  described,  “mock-up”  screen  shots  were 
produced  and  shown  to  the  focus  group.  In  general,  the  design  and  content  were  approved  by  the 
group,  although  some  specific  changes  were  recommended.  Access  to  the  CITT  help  desk  was 
removed;  an  advanced  search  function  was  added  that  searches  both  indices  and  complete  web 
pages;  and  the  proposed  use  of  a  bulletin  board  was  removed. 

Once  development  requirements  were  sufficiently  identified,  development  of  the  site 
using  commercial-off-the-shelf  software,  such  as  Front  Page,  InterDev,  and  SQL  Server,  was 
initiated  and  continued  throughout  the  remainder  of  the  project. 
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Formative  Evaluation 


Formative  evaluation  (FE)  was  completed  primarily  in  conjunction  with  the  limited 
fielding  of  CUT  1.1.  This  section  describes  those  efforts  as  well  as  the  limited  evaluation  that 
was  completed  for  CITT  2.0. 

CITT  1.1 


There  were  three  primary  objectives  for  FE  of  CITT  1.1: 

1.  To  determine  whether  commanders  and  other  unit  trainers  could  select,  modify,  and 
develop  CCTT  exercises  for  conventional  and  FBCB2-equipped  platoons  and  company  teams  on 
the  PI,  P2,  and  P3  TDBs. 

2.  To  determine  whether  commanders  and  other  unit  trainers  could  access,  modify, 
develop,  and  print  required  training  support  materials. 

3.  To  determine  whether  the  unit  could  fully  execute,  in  conjunction  with  CCTT  site 
personnel,  exercises  selected,  modified,  or  created. 

To  test  completion  of  these  objectives,  the  team  requested,  through  the  COR,  support 
from  units  at  Fort  Benning  and  Fort  Hood,  and  from  an  Army  National  Guard  (ARNG)  unit. 
Units  needed  to  be  available  for  participation  during  the  June-October  1999  time  frame,  and  the 
Fort  Hood  unit  also  needed  to  be  involved  with  the  ongoing  FBCB2  testing.  A  timeline  for  FE 
with  each  unit  was  developed  and  is  shown  in  Figure  2.  The  FE  with  each  unit  was  designed  to 
last  approximately  two  months  beginning  with  identification  of  the  unit  and  ending  when  the  unit 
ran  the  exercises  developed  with  CITT  at  the  CCTT  site. 


Figure  2.  Timeline  in  weeks  for  conducting  formative  evaluation  with  a  typical  unit. 

Specific  units  were  identified  at  Fort  Benning  and  Fort  Hood.  The  unit  at  Fort  Hood  was 
also  supporting  FBCB2  testing.  Although  much  effort  was  directed  at  identifying  an  ARNG 
unit,  due  to  scheduling  challenges,  no  ARNG  units  were  included  in  FE  limiting  the 
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generalizability  of  findings  to  Active  units.  Table  1  shows  the  schedule  of  FE  events  for  the 
units  assessed. 

Initial  user  training  consisted  of  approximately  four  hours  of  demonstration  and  hands-on 
training  at  the  unit’s  site  with  a  designated  point  of  contact  (POC).  The  unit  was  provided  a 
CITT  1.1  system  as  well  as  documentation  consisting  of  two  Getting  Started  Manuals  (User 
Getting  Started  and  Administrator  Getting  Started)  and  a  CITT  1.1  User’s  Manual.  The  CITT 
Team  member(s)  presenting  the  initial  training  made  it  clear  that  the  POC  was  responsible  for 
training  other  unit  personnel  who  would  use  CITT  including  those  who  would  participate  in 
additional  FE  activities.  They  also  informed  the  POC  that  a  CITT  “help  desk”  was  available 
either  via  toll-free  telephone  or  e-mail  to  respond  to  any  questions  or  concerns  users  might  have 
while  using  CITT. 

Table  1 

Schedule  of  Formative  Evaluation  Events 


Fort  Benning 

Fort  Hood 

Initial  User  Training 

30  June  1999 

3  August  1999 

CITT  Usability  Assessment 

19  -  22  July  1999 

12-13  August  1999  and 

16-18  August  1999 

CCTT  Build  and  Proof 

NA 

20  August  -  28  September  1999 

Exercise  Execution 

NA 

29  -  30  September  1999 

CITT  usability  assessment  consisted  of  observation  of  unit  personnel  using  CITT  to 
produce  TSPs.  Usability  assessment  corresponds  with  “unit  develops  exercises”  in  Figure  2. 

The  SOW  specified  that  this  phase  of  FE  would  consist  primarily  of  passive  observation,  and 
data  collection  forms  similar  to  those  used  in  FE  of  CITT  1.0  were  produced  (see  Appendix  B). 
However,  during  actual  assessment,  the  team  members  who  were  acting  as  observers  were  forced 
to  assume  a  more  active  role.  In  the  case  of  the  Fort  Benning  assessment,  the  CITT  system  had 
been  connected  to  the  unit’s  Local  Area  Network  (LAN),  contrary  to  instructions  from  the  CITT 
team,  which  in  turn  disabled  some  features,  particularly  the  map  and  overlay  tools.  The 
observers  had  to  produce  a  work-around  which  restored  nearly  all  functionality.  The  observation 
at  Fort  Benning  occurred  as  planned  with  three  members  of  the  unit  (the  brigade  master  gunner 
and  two  battalion  master  gunners)  acting  as  participants.  The  brigade  master  gunner  was  very 
familiar  with  the  structured  training  process  and  had  little  difficulty  using  CITT  to  develop  a 
usable  exercise;  the  battalion  master  gunners  had  little  familiarity  with  structured  training  and 
required  assistance  from  the  observers  and  the  brigade  master  gunner  rather  than  risk  having 
them  produce  unusable  exercises.  At  Fort  Hood,  one  unit  member  (the  assistant  operations  and 
training  officer)  served  as  participant  for  the  entire  usability  assessment.  However,  the  unit  was 
under  substantial  pressure  to  produce  exercises  for  use  in  the  test  of  FBCB2  making  it  imperative 
that  they  produce  usable  exercises.  Under  this  circumstance,  passive  observation  was  not 
practical  and  considerable  assistance  was  provided  by  the  observers.  The  implications  of 
attempting  to  conduct  FE  with  operational  units  using  passive  observation  methods  will  be 
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discussed  below.  Following  observation  of  the  participant  while  using  CITT,  interviews  were 
conducted  using  the  structured  interview  form  contained  in  Appendix  B. 

The  CCTT  build  and  proof  activity  occurred  only  at  Fort  Hood.  This  corresponds  to 
“Site  builds  and  tests  exercises”  in  Figure  2.  In  the  case  of  the  Fort  Benning  unit,  timing 
problems  occurred  between  the  unit’s  scheduled  CCTT  rotation  and  the  availability  of  site 
personnel  to  build  the  exercises.  This  was  further  exacerbated  by  part  of  the  unit  being  deployed 
to  Egypt,  and,  therefore,  unavailable  to  run  exercises.  At  Fort  Hood,  the  exercises  were  built  and 
tested  at  the  site,  and,  as  anticipated,  minor  problems  were  discovered  and  corrected.  This 
included  such  things  as  missing  radio  frequencies,  bumper  numbers  that  changed  from  original 
settings,  etc.  Members  of  the  development  team  did  not  observe  this  phase  of  FE,  although 
communication  between  the  site  and  the  team  was  occurring  throughout  the  process. 

Exercise  execution  (“Unit  runs  exercises  in  CCTT”  in  Figure  2)  corresponded  with  the 
Fort  Hood  unit’s  participation  in  the  FBCB2  test.  Two  members  of  the  CITT  team  observed  this 
phase  of  FE  and  collected  data  using  the  observation  and  interview/debriefing  forms  contained 
in  Appendix  C.  Although  the  unit  commander  made  some  initial  modifications  to  the  CITT- 
developed  exercises,  the  unit  did  conduct  the  company/team  movement  to  contact  exercises 
developed  during  the  CITT  usability  assessment  phase  of  FE  a  total  of  three  times  over  two  days. 
The  CITT-developed  exercises  did  initialize  properly  and  ran  sufficiently  well  to  allow  the  unit 
to  conduct  its  training. 

Additional  evaluation,  not  originally  included  in  the  Research  Program  Plan,  was 
conducted  at  two  locations  at  Fort  Knox.  Personnel  from  the  Advanced  Non-Commissioned 
Officers  Course  (ANCOC)  requested  a  CITT  to  determine  its  applicability  to  their  training. 
Shortly  thereafter,  personnel  from  the  Armor  Captains  Career  Course  (ACCC)  requested  a 
system  for  similar  reasons.  Systems  were  delivered  along  with  the  initial  user  training  that  had 
been  provided  at  Fort  Hood  and  Fort  Benning.  Personnel  were  invited  to  use  CITT  as  they 
deemed  appropriate  but  were  requested  to  notify  the  CITT  team  of  any  planned  use  so  that 
observations  could  be  conducted.  In  fact,  only  one  observation  was  conducted  at  ANCOC;  two 
were  conducted  at  ACCC.  The  pressures  of  everyday  activities  limited  the  time  the  users  were 
able  to  access  and  use  CITT.  Implications  of  conducting  evaluation  while  participants  conduct 
their  normal  operations  will  be  discussed  below. 

The  final  evaluation  activity  that  occurred  with  CITT  1.1  involved  locating  CITTs  at 
three  CCTT  sites  (Fort  Hood,  Fort  Benning,  and  Fort  Knox)  and  at  the  TRADOC  System 
Manager  CATT  and  Program  Manager  CATT  offices  to  provide  an  opportunity  for  anyone 
wishing  to  use  them  to  do  so.  Thus,  a  total  of  nine  CITT  1.1  systems  were  fielded  to  support  FE. 
An  embedded  questionnaire  is  included  in  CITT  and  is  offered  to  the  user  each  time  he  or  she 
makes  a  normal  exit.  There  is,  however,  no  practical  way  to  force  users  to  complete  the 
questionnaire.  In  late  November,  the  POCs  at  each  location  were  instructed  to  copy  the  CITT 
database  file  to  a  large-capacity  disk  provided  to  them  and  to  forward  it  to  the  CITT  team.  All 
implementation  sites,  except  for  the  CCTT  site  at  Fort  Benning,  complied.  The  Fort  Benning 
CCTT  site  had  a  hardware  failure  with  the  disk  drive  and  was  unable  to  copy  the  file.  The 
database  file  contained  all  completed  user  surveys  as  well  as  copies  of  TSPs  produced.  Analysis 
of  the  returned  files  indicated  that  no  surveys  were  completed  other  than  those  completed  during 
the  observation  sessions  described  previously,  and  that  no  additional  completed  exercises  had 
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been  produced.  The  present  study  offers  no  definitive  explanation  for  this  finding,  however,  it 
could  be  speculated  that  users  currently  view  CITT  as  research  and  development  effort,  and  that 
it  will  need  to  become  institutionalized  within  the  training  culture  before  it  will  be  widely  used. 

cm  2.0 

Because  work  on  the  CITT  2.0  development  continued  up  to  the  end  of  the  project,  there 
was  no  opportunity  to  conduct  FE  beyond  internal  testing  by  members  of  the  CITT  team.  Even 
this  testing  was  limited.  As  components  of  CITT  2.0  were  completed,  they  were  placed  on  a  test 
machine  and  members  of  the  team  exercised  them  performing  all  reasonable  actions  that  users 
are  likely  to  perform.  This  was  done  to  determine  if  the  software  worked  as  necessary  and  to 
ensure  that  the  system  did  not  “break.”  All  problems  and  defects  discovered  were  reported  to  the 
developers  and  appropriate  modifications  were  made. 

Implementation  and  Fielding 

The  research  methodology  for  this  phase  of  the  project  was  directed  at  producing  an 
implementation  strategy  and  fielding  plan/methods  for  the  CITT.  This  required  consideration  of 
the  full  range  of  individuals  who  will  use  the  CITT  including  remote  users  and  system 
administrators.  In  addition,  the  plan  needed  to  address  compatibility  of  the  CITT  with  Army 
training  information  management  systems  and  databases  as  well  as  the  fielding  and  sustainment 
of  the  CITT. 

Initial  plans  for  this  activity  as  presented  in  the  Research  Program  Plan  were  for  near- 
term  implementation  and  fielding  to  address  the  CITT  as  it  exists  at  the  end  of  the  CITT-2 
Project,  and  for  long-term  implementation  and  fielding  to  examine  the  integration  of  CITT  into 
the  Standard  Army  Training  System  (SATS).  Late  in  the  project,  the  team  realized  that  it  needed 
to  consider  near-,  mid-,  and  long-term  solutions  to  allow  for  incremental  development  of  an 
objective  system,  further  evaluation  and  refinement  of  the  CITT,  and  more  time  for  CITT  to  be 
configured  for  integration  into  SATS. 

Our  process  included  reviewing  the  strategy  and  plan  proposed  as  part  of  the  initial  CITT 
project,  reviewing  plans  for  similar  Army  training  systems  (i.e.,  Automated  Systems  Approach  to 
Training  [ASAT]),  and  investigating  emerging  Web  technologies  and  methods  for  using  such 
technologies. 

As  part  of  the  initial  CITT  project  we  proposed  three  options  for  such  a  plan  (Gossman  et 
al.,  1999).  They  were:  field  CITT  specifically  to  support  CCTT;  field  CITT  as  a  part  of  SATS 
and/or  ASAT;  and  develop  and  field  an  Integrated  Training  Tool  (environment  non-specific). 

The  first  option  recommended  that  CITT  be  fielded  as  a  stand-alone  system  (software  and 
hardware)  to  CCTT  sites  and  selected  units  to  ensure  that  trainers  could  access  information  on 
CCTT  training  and  have  available  an  existing  library  of  CCTT  exercise  TSPs.  This  was  intended 
as  a  short-term  solution.  The  second  option  recognized  a  longer-term  need  that  CITT  (or  any 
TSP  development  tool)  become  part  of  SATS  to  provide  trainers  with  one  integrated  software 
package  for  training  management  activities.  This  remains  the  optimum  long-term  solution.  The 
third  option  attended  to  the  need  for  a  distributed  CITT-like  system  that  supports  training 
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development  efforts  for  actions  in  all  training  environments  (live,  virtual,  and  constructive).  This 
option  also  remains  valid  and  was  considered  as  part  of  the  current  effort. 

Our  review  of  existing  similar  Army  systems  focused  on  the  architecture  of  AS  AT  which 
is  administered  by  the  Army  Training  Support  Center  (ATSC)  and  can  be  classified  as  a  client- 
server  system.  The  AS  AT  is  a  distributed  system  where  each  proponent  school  maintains  and 
works  on  its  own  database  (MTPs,  Field  Manuals,  TSPs  etc.)  but  shares  data  with  other  agencies 
as  required  via  the  Training  Module  Executive  Management  Information  System  (TEXMIS)  data 
repository.  Local  data  are  stored  on  a  common  server  and  updated  as  required  by  clients  who 
have  the  required  access  and  authority.  Once  the  data  are  ready  for  distribution  to  others,  they 
are  uploaded  from  the  server  to  TEXMIS  where  those  with  the  required  access  and  authority 
have  access  to  them.  Once  finalized  and  approved,  the  data  are  made  available  to  other  clients 
and  the  Army  at  large  via  the  Reimer  Digital  Library.  Likewise,  proponent  agencies  can 
download  data  from  TEXMIS  to  their  server  for  manipulation  as  required.  This  architecture 
stands  as  the  model  for  the  long-term  implementation  and  fielding  plan  described  later  and 
makes  effective  use  of  today’s  Web  and  client-server  technologies. 

We  also  recognized  it  was  important  to  investigate  emerging  Web  technologies  and 
methods  being  used  to  manage  distribution  of  data.  Midway  through  the  project  we  were 
fortunate  enough  to  be  offered  the  assistance  of  a  U.S.  Army  Reserve  Signal  Corps  Officer 
currently  managing  a  software  development  and  support  center. 

The  Reserve  officer  provided  a  valuable  additional  viewpoint  to  the  research  on  the 
appropriate  use  of  emerging  technologies.  He  agreed  to  design  a  server-based  architecture  that 
would  meet  the  long-term  requirements  for  a  distributed  system  akin  to  what  ATSC  currently 
uses  within  the  U.S.  Army  Training  and  Doctrine  Command  community.  This  effort  focused  on 
using  new  Web  technologies  to  provide  a  robust  system  that  allows  unit  trainers  access  to  up-to- 
date  training  information,  exercise  TSPs,  and  the  ability  to  modify  or  create  TSPs  as  required. 

The  requirement  for  users  to  have  ready  access  to  information  and  training  materials  that  support 
the  development  and  execution  of  structured  training  that  meets  their  specific  training  needs 
remained  at  the  core  of  this  process.  For  the  near-  and  mid-term  plans,  a  CITT  for  CCTT 
training  was  considered;  the  long-term  plan  addressed  a  system  architecture  that  is  not 
simulation-specific. 

Our  research  focused  on  expansion  and  refinement  of  each  of  the  three  options  proposed 
in  the  initial  CITT  project.  We  considered  the  current  plans  for  CITT  development,  the  current 
plans  for  CCTT  fielding,  units’  abilities  to  access  information  via  the  Internet,  along  with  those 
options  which  are  feasible  considering  future  plans  being  considered  by  affected  agencies 
(specifically  PM-CATT  and  ATSC).  Under  this  broad  set  of  considerations,  we  addressed  the 
following  specific  questions: 

1 .  Where  and  to  whom  should  CITT  be  fielded? 

2.  What  is  the  need  for  initial  user  training;  what  is  it;  and  who  gets  it? 

3.  What  is  the  need  for  a  “Help  Desk?” 
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4.  Who  should  maintain  the  Web  Site  and  where  should  it  be  hosted? 

5.  What  skills  and  numbers  of  personnel  are  required  to  maintain  the  CITT? 

6.  What  is  the  process  for  exercise  TSP  development,  approval,  and  distribution? 

Results  and  recommendations  from  this  research  task  are  presented  below  along  with 
results  from  all  research  activities. 


Results 

This  section  focuses  on  the  outcomes  and  products  of  the  research  activities  described 

above. 


Design  the  Enhanced  CITT 

The  major  outcome  of  the  design  activities  is  a  complete  description  of  the  enhanced 
CITT  presented  in  the  form  of  EDEFO,  IDEF3,  and  Dynamic  Data  Exchange  diagrams  and  tables. 
The  design  reflects  how  the  enhanced  CITT  will  function  and  supports  its  continued 
development.  The  design  is  fully  consistent  with  existing  Army  training  systems  and  those 
expected  to  exist  in  the  near  future.  Examples  of  top  level  design  in  the  form  of  two  IDEFO 
diagrams-an  Input,  Control,  Outcome,  Mechanism  diagram  and  a  node  tree  diagram  are  shown 
in  Figures  3  and  4  respectively. 
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Figure  3.  Enhanced  CITT  design  top  level  Input,  Control,  Outcome,  Mechanism. 
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Figure  4.  Enhanced  CITT  design  top-level  node  tree  diagram. 

The  complete  design  is  contained  in  an  ARI  Research  Note  (Dannemiller  &  Gossman,  in 
preparation).  This  Research  Note  has  two  primary  purposes:  the  short-term  purpose  is  to 
describe  in  detail  the  CITT  design  for  presentation  to  and  review  by  appropriate  agencies  such  as 
ATSC;  the  long-term  purpose  is  to  capture  the  complete  design  such  that  an  application  based 
upon  it  can  be  built  at  a  later  time.  Descriptions  of  the  functions  and  use  of  the  different  types  of 
design  methods  along  with  the  complete  set  of  design  documents  for  the  CITT  are  included  in 
the  Research  Note. 


Develop  Second-Generation  CITT  Prototype 


CITT  1.1 

As  described  above,  the  CITT  1.1  was  produced  in  a  relatively  short  time  for  limited 
fielding  to  support  FE,  and,  as  such,  the  changes  and  modifications  were  primarily  related  to 
FBCB2  and  the  PI  TDB. 

Overall,  the  application  database  grew  approximately  40%.  Five  MTPs  related  to 
digitally  equipped  units  were  added  as  were  four  exercises  (two  digital  and  two  on  the  PI  TDB.) 
Thus,  CITT  1.1  shipped  with  a  total  of  39  exercises.  The  application  ran  substantially  faster  than 
CITT  1.0  due  to  the  fact  that  when  an  exercise  was  accessed,  icons  depicting  documents  and 
maps  were  loaded  rather  than  the  actual  document  or  map.  This  significantly  reduced  record 
size.  It  also  shipped  with  an  updated  IO  and  Help  and  with  guidelines  for  converting 
conventional  exercises  to  digital  and  vice  versa. 
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CITT  1.1  was  installed  on  PM-provided  computers  and  fielded  at  nine  locations  as 
described  above  under  FE. 

crrT2.o 

In  keeping  with  the  CITT  2.0  development  goals  of  simplifying  the  user  interface  and  the 
TSP  development  process,  a  number  of  modifications  were  made  and/or  functions  were  added 
based  on  the  input  of  the  focus  group: 

1 .  The  Training  Event  Diagram  was  renamed  the  Exercise  Concept  Sketch. 

2.  The  Overlay  was  renamed  the  Mission  Concept  Sketch. 

3.  The  user  was  given  the  ability  to  modify  the  conditions  and  standards  for  each  task. 

4.  “Why”  buttons  were  added  to  most  screens. 

The  first  two  modifications  above  were  recommended  by  the  focus  group  to  simplify  the 
terminology  used  in  CITT.  The  third  was  recommended  to  provide  more  flexibility  to  the  trainer 
in  designing  an  exercise.  The  fourth  was  a  result  of  the  group’s  discussion  regarding  adding 
more  cognitive  information.  That  is,  not  only  does  the  CITT  lead  the  user  through  the  TSP 
development  process,  it  also  provides  the  user  with  information  on  why  and  how  components  of 
the  TSP  fit  into  the  overall  structured  training  process. 

Figures  5  and  6  display  some  of  the  features  and  characteristics  of  the  CITT  2.0  at  the 
completion  of  the  project.  Figure  5  is  the  opening  screen  which  allows  the  user  to  access  the 
various  CITT  functions  including  the  CITT  Overview,  the  Learn  About  CCTT  Information 
Overview,  the  CCTT  Site  Tools,  and  the  Produce  Training  Materials.  It  allows  access  to  the 
CITT  Administrative  Tools  which  provide  the  functionality  required  to  transfer  or  upload 
exercise  TSPs  to  a  central  repository.  And  it  includes  access  to  the  embedded  User  Survey 
which  allows  the  user  to  provide  feedback  to  the  development  team  if  he  or  she  so  desires. 

Figure  6  is  an  example  of  one  screen  taken  from  the  Create  Exercise  TSP  branch  of  the 
Produce  Training  Materials  function.  This  specific  screen  illustrates  how  the  user  establishes 
initial  settings  for  a  mission  within  which  an  exercise  is  developed.  Of  significance  is  the  fact 
that  this,  and  most  other  screens  in  the  Produce  Training  Materials  function  includes  a  “Why” 
button.  When  users  access  the  “Why”  button,  they  are  provided  with  a  detailed  explanation  of 
the  “cognitive”  aspects  of  TSP  development.  That  is,  they  are  given  information  on  how  an 
activity  fits  within  and  supports  structured  training  in  order  to  provide  them  with  a  basis  for  what 
they  are  doing  as  opposed  to  simply  helping  them  complete  the  activity. 
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Figure  6.  An  example  of  a  Produce  Training  Materials  screen. 
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CUT  Web  Site 


The  results  of  the  Web  Site  development  activities  are  best  viewed  at  the  site  itself— 
www.cittcctt.org.  Development  of  the  site  continues  on  an  on-going  basis.  However,  there  are  a 
number  of  specific  outcomes  of  the  development  activities  that  should  be  described  in  this  report. 

First,  it  should  be  noted  that  there  was  nearly  a  two-month  delay  on  starting  development 
of  the  site  due  to  uncertainty  regarding  what  would  be  in  the  CUT  database  and  in  CEU.  An 
important  lesson  learned  from  this  project  is  that  final  decisions  on  the  database  structure  must 
occur  for  web-based  development  to  occur. 

For  the  most  part,  the  Web  Site  provides  all  of  the  planned  functionality.  There  are, 
however,  some  limitations.  The  View  an  Exercise  function  allows  the  user  to  view  only  three 
components  of  the  TSP:  the  Exercise  Description,  the  Exercise  Concept  Sketch;  and  the  Event 
Description  including  exercise  tasks.  The  user  can  download  the  entire  TSP,  but  only  these 
components  can  be  viewed  from  the  Web  Site.  In  addition,  the  user  can  not  print  from  the  site. 

On  the  other  hand,  the  user  can  access  the  complete  Learn  About  CCTT  module  of  CUT; 
there  is  access  to  the  CUT  videos;  there  is  a  fully  functioning  newsgroup  capability;  TSPs  are 
available  for  review  and  download;  the  upload  TSP  function  operates;  there  are  links  to  other 
relevant  Army  Internet  sites;  and  there  are  pages  containing  information  for  each  of  the  CCTT 
sites.  The  site  includes  an  expanded  search  capability  which  allows  users  to  search  entire  web 
pages,  not  just  the  keyword  indices,  as  well  as  advanced  navigation  capabilities  which  facilitate 
its  use.  These  functions  are  illustrated  in  Figure  7. 


Formative  Evaluation 


CITT  1.1 


Help  Desk.  No  calls  or  e-mail  were  received  via  the  CITT  help  desk  during  the  period  in 
which  CITT  1.1  was  fielded.  It  is  unclear  why  this  result  occurred.  It  is  possible  that  there  were 
no  difficulties  encountered  by  users;  it  is  also  possible  that  there  were  simply  few  people  actually 
using  CITT.  This  is  consistent  with  results  from  the  CITT  data  retrieval  that  occurred  in 
December  as  described  below. 

CITT  Usability  Assessment.  As  described  above,  usability  assessment  occurred  at  Fort 
Hood  and  at  Fort  Benning.  Because  of  substantial  differences  in  the  way  assessment  occurred  at 
each  site,  they  will  be  discussed  separately. 

At  Fort  Benning,  three  participants  were  observed  over  a  four-day  period.  All 
participants  were  using  CITT  to  develop  platoon  gunnery  exercises.  As  the  first  participant 
began  developing  an  exercise,  it  became  clear  to  the  observers  that  CITT  was  not  operating 
correctly.  Specifically,  the  map  and  overlay  tools  were  not  functioning  at  all.  The  observers 
discovered  that  the  CITT  machine  had  been  connected  to  the  unit’s  LAN,  and,  because  the  map 
and  overlay  tools  are  installed  directly  to  the  WindowsNT  administrator  identification  (ID),  they 
would  not  function  with  any  other  user  ID.  Moreover,  as  would  be  expected  in  most  network 
environments,  the  network  administrator  had  removed  the  administrator  ID  from  the  machine. 
The  remainder  of  the  first  morning  was  spent  correcting  this  problem  and  getting  the  map  and 
overlay  tools  to  function.  Once  corrected,  usability  assessment  resumed. 

For  the  most  part,  passive  observation  was  employed,  although,  if  a  participant  requested 
help  and  had  been  unable  to  determine  an  appropriate  action  himself,  assistance  was  provided. 
Also,  if  it  became  clear  to  the  observers  that  a  participant  was  having  difficulty  or  was  heading  in 
a  direction  that  would  prove  futile,  assistance  was  offered.  In  this  instance,  the  three  participants 
were  quite  different  in  terms  of  their  understanding  of  structured  training  concepts  and  principles 
as  well  as  their  level  of  computer  literacy.  The  participant  who  understood  structured  training 
quite  well  and  was  highly  computer-literate  required  little  assistance,  and  the  assistance  required 
was  primarily  when  either  the  map  and  overlay  tools  or  the  print  function  did  not  work  properly. 
A  second  participant  was  computer-literate  but  had  little  concept  of  structured  training.  After 
coaching  on  structured  training,  he  was  able  to  use  CITT  with  minimal  assistance.  The  third 
participant  was  neither  computer-literate  nor  had  a  good  understanding  of  structured  training 
concepts.  This  participant  required  extensive  coaching  throughout  the  assessment.  Results  from 
this  limited  sample  suggest  that  the  more  versed  in  the  concepts  of  structured  training  a 
participant  is,  the  more  efficient  and  effective  his  use  of  CITT  will  be. 

Another  aspect  of  the  assessment  at  Fort  Benning  concerns  the  environment  in  which  FE 
occurred.  The  CITT  machine  had  been  located  at  the  brigade  master  gunner’s  desk  in  the 
brigade  operations  shop.  All  three  participants  used  CITT  at  that  location.  The  environment  was 
very  noisy  and  participants  were  frequently  distracted  from  their  use  of  CITT  by  telephone  calls, 
questions  and  requests  from  other  unit  personnel,  and  so  forth.  While  this  may  be  similar  to  the 
environment  in  which  CITT  will  ultimately  be  used,  it  was  a  difficult  environment  in  which  to 
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conduct  FE.  A  more  controlled  environment  would  likely  produce  better  results  especially  in  the 
early  phases  of  system  development  and  FE. 

All  participants  successfully  developed  at  least  one  exercise,  although  one  did  not  enter 
the  complete  Operations  Order  (OPORD).  In  general,  they  provided  positive  reactions  to  CUT 
both  on  the  embedded  survey  and  during  the  interview/debriefing.  All  stated  that  CUT  would  be 
a  valuable  tool  for  units  as  they  prepared  for  training  in  CCTT. 

Following  completion  of  usability  assessment,  members  of  the  CUT  team  who  had 
participated  as  observers  met  to  debrief  and  to  produce  a  list  of  findings  and/or  implications  for 
CUT  development.  Appendix  D  contains  a  list  of  36  findings  (including  those  from  ANCOC 
described  below).  This  list  of  findings  was  subsequently  used  as  the  basis  for  a  CUT  team 
synchronization  meeting  to  discuss  implications  and  implementation  in  future  versions  of  CUT. 
Some  of  the  more  significant  findings  include: 

1.  CUT  needs  to  work  properly  even  when  the  host  machine  is  attached  to  a  network. 
This  is  especially  important  in  light  of  implementation  and  fielding  recommendations  listed 
below,  particularly  as  related  to  the  use  of  the  CUT  Web  Site  as  a  TSP  repository. 

2.  Coordination  between  the  CUT  team  and  the  CEU  development  team  is  crucial  to 
ensuring  that  CEU  functions  properly  within  CUT.  In  addition,  CEU-specific  help  needs  to  be 
written;  if  the  CEU  team  is  not  producing  help,  the  CUT  team  needs  to. 

3.  The  print  function  of  CUT  needs  to  be  made  efficient.  Currently,  CUT  employs  the 
print  preview  function  of  Microsoft  Word  in  order  to  print  parts  of  the  TSP.  This  is  confusing  to 
the  user  and  inefficient.  (In  fact,  a  patch  to  CUT  was  produced  that  corrected  this  problem  and 
was  distributed  to  all  CUT  1 . 1  sites.) 

4.  CUT  needs  to  be  fielded  with  a  formalized  training  process  unless  or  until  structured 
training  concepts  and  principles  become  part  of  the  basic  or  advanced  courses  for  officers  and 
non-commissioned  officers. 

5.  Current  navigation  in  the  Create  module  is  confusing.  Users  may  think  they  have 
completed  exercise  creation  when,  in  fact,  they  have  completed  only  the  exercise  outlines. 

6.  In  the  Create  module,  the  current  default  is  “Complete”  for  each  exercise  created.  If  a 
user  exits  CUT  prior  to  actually  completing  the  exercise  and  then  returns,  he  is  unable  to 
complete  the  exercise  using  the  create  process.  He  is  forced  to  use  the  Modify  module.  The 
default  needs  to  be  changed  so  that  the  user  can  use  the  Create  module  throughout  the 
development  process. 

Usability  assessment  at  Fort  Hood  involved  one  participant  and  occurred  in  two  sessions- 
the  first  lasted  two  days;  the  second  lasted  three  days.  The  assessment  was  planned  to  coincide 
with  the  FBCB2  test  and  required  that  exercises  be  developed  and  delivered  to  the  CCTT  site  by 
18  August  1999.  This  resulted  in  a  change  of  focus  from  collecting  FE  data  to  ensuring  that  the 
unit  developed  exercises  that  fit  their  training  needs.  The  CUT  team  observers  became  active 
participants  in  exercise  development  rather  than  passive  observers  limiting  conclusions  that  can 
be  drawn  from  the  sessions. 
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The  end  result  of  usability  assessment  with  the  Fort  Hood  unit  was  that  three  exercises 
were  developed  to  be  used  in  the  FBCB2  test  which  corresponded  to  the  exercise  execution 
phase  of  CITT  FE.  The  participant,  as  well  as  other  unit  personnel,  provided  favorable 
impressions  of  CITT,  and,  in  fact,  indicated  a  desire  to  have  CITT  2.0  to  assist  in  preparation  of 
the  unit’s  scheduled  February  CCTT  training.  In  the  survey  and  in  debriefing,  the  participant 
also  mentioned  the  desirability  of  having  CITT  produce  a  file  that  could  be  used  to  initialize  an 
exercise  at  the  CCTT  site  which  is  the  ultimate  intent  of  CEIT. 

As  with  Fort  Benning,  CITT  team  observers  debriefed  the  sessions  at  Fort  Hood  and 
produced  the  list  of  findings  provided  in  Appendix  E.  These  were  presented  to  the  entire  team 
and  decisions  were  made  on  how,  or  if,  they  would  be  handled  in  future  versions  of  CITT. 
Representative  findings  from  Fort  Hood  include  the  following: 

1.  The  unit  had  connected  the  CITT  machine  to  their  LAN  resulting  in  the  same 
problems  as  those  encountered  at  Fort  Benning. 

2.  The  CITT  install  failed  because  the  machine  did  not  have  Microsoft  Agent  installed. 

3.  The  quality  of  the  videos  used  in  the  Tank  and  Mechanized  Infantry  Platoon 
Demonstrations  of  Performance  is  very  poor  when  using  a  laptop  computer.  This  is  probably 
due  to  the  lower  video  memory  on  the  laptop. 

4.  The  CITT  uninstall  function  does  not  completely  remove  CITT. 

Two  locations  at  Fort  Knox  provided  the  final  usability  assessment  data  for  the  project: 
ANCOC  and  ACCC.  Two  sessions  were  conducted  at  ANCOC;  one  session  was  conducted  at 
ACCC.  However,  unlike  Fort  Benning  and  Fort  Hood,  the  sessions  were  much  shorter  (less  than 
two  hours)  and  were  done  at  the  discretion  of  the  participants.  The  participants  were  not 
debriefed,  and  neither  completed  the  embedded  survey.  The  findings  for  ANCOC  are  included 
in  Appendix  D  along  with  those  from  Fort  Benning;  the  findings  from  ACCC  are  included  in 
Appendix  E  along  with  those  from  Fort  Hood. 

CCTT  Exercise  Build  and  Proof.  The  exercise  build  and  proof  phase  of  FE  was 
conducted  only  at  Fort  Hood.  (For  reasons  outside  the  scope  of  this  project,  exercises  developed 
at  Fort  Benning  were  not  built  at  the  CCTT  site.  Part  of  the  unit  was  deployed  to  Egypt,  and  the 
brigade  master  gunner  was  assigned  to  temporary  duty  at  the  Fort  Knox  ANCOC.)  Members  of 
the  CITT  team  did  not  observe  the  build/proof  activities  at  Fort  Hood;  however,  they  did 
interview  site  personnel  several  weeks  after  the  exercises  were  built.  The  report  at  that  time  was 
that  the  information  necessary  to  build  the  exercises  was  contained  in  the  TSPs  received  from  the 
unit,  but  that  it  was  difficult  to  find  unless  one  is  very  familiar  with  the  TSP  materials.  The  TSP 
does  not  contain  a  statement  of  exercise  focus  or  intent,  and  there  were  problems  with  the 
commo  lists.  Site  personnel  reported  that  they  thought  the  map  and  overlay  tools  could  be  more 
user  friendly.  They  also  reported  that  units  do  not  participate  in  the  build/proof  process. 

Exercise  Execution.  Exercise  execution  was  conducted  at  Fort  Hood  with  two  members 
of  the  CITT  team  serving  as  observers.  The  exercise  that  had  been  developed  using  CITT  was  a 
Company/Team  Movement  to  Contact  exercise  that  included  all  friendly  and  enemy  semi- 
automated  forces  required  for  a  complete  mission  set.  However,  on  the  first  morning,  the 
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company  commander  decided  to  abandon  the  original  plan  and  have  his  platoons  conduct 
platoon-level  tactical  movement  from  an  assembly  area  (AA)  to  the  line  of  departure.  On  the 
first  afternoon  and  all  of  the  second  day,  the  unit  ran  the  CITT-developed  movement  to  contact 
exercise  three  times.  For  each  running,  the  unit  started  in  the  AA  and  conducted  a  movement  to 
contact  until  the  training  company  was  combat  ineffective.  The  battalion  operations  officer 
served  as  the  observer/controller  (O/C). 

The  CITT-developed  exercises  initialized  properly  and  ran  well  enough  to  allow  the  unit 
to  conduct  its  training.  On  the  other  hand,  there  were  a  number  of  difficulties,  only  some  of 
which  are  attributable  to  CITT.  The  unit  had  not  written  an  OPORD,  nor  did  the  unit  use 
exercise  graphics  in  preparing  to  run  the  exercise.  The  O/C  did  not  use  the  after  action  review 
(AAR)  sheets  included  in  the  TSP,  opting  instead  to  use  index  cards.  There  was  a  lack  of  clear 
instructions  on  assigning  and  using  the  FBCB2  workstations  at  the  AAR  workstation.  The 
company  commander  and  operations  officer  were  unaware  that  the  FBCB2  workstations  at  the 
AAR  worked  and  could  be  used  to  monitor  and  affect  the  unit’s  performance  during  the  exercise. 
During  the  TSP  development  process  in  CITT,  the  user  needs  to  be  able  to  designate  vehicle 
assignments  for  the  FBCB2  workstations  at  the  AAR. 

Follow-up  interviews  of  unit  personnel  indicated  that  there  was  some  confusion  regarding 
CITT  replacing  face-to-face  meetings  with  site  personnel.  Unit  personnel  had  operated  under  the 
assumption  that  no  coordination  between  the  unit  and  site  was  required  once  the  paper  TSP  was 
delivered  to  the  site.  Initial  training  when  CITT  is  delivered  needs  to  clarify  the  unit-site 
communication  and  interaction.  There  was  concern  expressed  by  unit  personnel  that  units  do  not 
have  sufficient  time  to  create  exercises.  They  recommended  that  CITT  include  a  broader,  more 
diverse  base  set  of  exercises  for  units  to  select  from  and  modify  if  necessary.  This  library  of 
exercises  should  contain  exercises  for  different  types  of  units  on  different  terrain  fighting  against 
the  appropriate  OPFOR.  The  OPFOR  should  be  theater-specific  and  fight  using  the  appropriate 
doctrine. 

A  final  observation  from  the  CITT  team:  the  unit  and  site  personnel  never  conducted  a 
joint  proofing  of  the  exercise  which  would  have  shown  whether  timings,  locations,  and  points  of 
initial  contact  were  correct.  When  the  unit  ran  the  exercise,  it  became  clear  that  it  was  written  at 
a  “run”  level.  The  enemy  met  the  unit  at  a  location  where  they  had  the  advantage,  quickly  closed 
its  second  and  third  echelon  formations  on  the  training  unit  and  had  artillery  pre-positioned  to 
attack.  The  O/C  was  unaware  that  he  could  change  the  exercise  as  it  was  running,  which  led  to 
the  unit  conducting  a  difficult  exercise  first  and  being  defeated.  After  being  told  he  could 
modify  the  exercise,  the  O/C  had  the  unit  rerun  it  twice  under  less  difficult  conditions.  The 
actual  progression  was  “run-walk-crawl.”  This  difficulty  would  have  been  avoided  had  the  unit 
and  site  personnel  jointly  proofed  the  exercise. 

CITT  2.0 


As  described  previously,  FE  conducted  on  CITT  2.0  consisted  only  of  internal  testing. 
All  problems  and  defects  found  were  reported  to  the  developers.  It  is  anticipated  that  a  future 
project  will  have  FE  of  CITT  2.0  as  one  of  its  major  foci. 
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Implementation  and  Fielding 


The  key  findings  that  impact  a  comprehensive  implementation  strategy  and  fielding  plan 
suggest  that  an  incremental  or  staged  approach  to  the  fielding  of  CITT  is  appropriate.  The 
following  is  a  list  of  findings  that  resulted  from  our  review.  This  includes  the  notion  that  CITT 
can  and  should  initially  be  fielded  as  a  stand-alone  system  in  support  of  CCTT  training,  and  that 
a  dedicated  training  support  system  needs  to  be  developed  to  support  unit  trainers  as  they  design 
and  develop  training  for  all  environments. 

1.  The  CITT  2.0  and  the  new  CITT  Web  Site  have  not  been  fully  evaluated,  so  further 
evaluation  and  refinement  are  needed  before  putting  CITT  into  production.  This  requires  another 
round  of  limited  fielding  for  evaluation  purposes. 

2.  For  the  time  being  CITT  should  remain  focused  on  CCTT  training. 

3.  Not  enough  research  has  been  done  to  identify  how  CITT  should  be  integrated  into 
SATS;  therefore,  CITT  should  initially  be  fielded  as  a  separate  system  (CITT  2.0  for  PCs  and  the 
CITT  Web  Site). 

4.  The  CITT  fielding  should  be  closely  aligned  with  CCTT  fielding  plans. 

5.  At  the  end  of  the  next  round  of  evaluations,  CITT  should  be  moved  from  R&D  to  an 
agency  with  the  resources  and  skills  to  implement  full  fielding,  sustainment,  and  future 
development. 

6.  Future  systems  should  take  full  advantage  of  emerging  Web  technologies  to  provide 
unit  trainers  even  more  ready  access  to  CITT  functions. 

7.  Operational  units  need  a  training  support  system  (hardware  and  software)  similar  to 
ASAT  to  fully  support  training  development  in  the  field. 

8.  CCTT  and  CITT  initial  training  should  become  part  of  institutional  programs  of 
instruction  to  help  ensure  that  unit  leaders  know  how  to  make  the  best  possible  use  of  the  CCTT. 

With  these  findings  in  mind,  the  team  developed  a  three-stage  strategy:  near-term  - 
March  2000  to  January  2001;  mid-term  -  January  2001  to  December  2001;  and  long-term  - 
January  2002  and  onward.  Following  this  staged  strategy  will  support  the  immediate  R&D  need 
and  allow  for  future  development  efforts  that  include  integration  with  existing  Army  Training 
Information  Management  Program  (ATIMP)  systems.  Each  stage  includes  fielding, 
implementation,  and  staffing  recommendations. 

Near-Term 


In  the  near-term  CITT  should  be  fielded  to  support  further  evaluation  and  refinements  to 
enable  the  development  of  a  fully  fieldable  system.  To  support  this  plan  CITT  2.0  software 
should  be  installed  at  all  existing  CCTT  locations.  A  laptop  computer  with  CITT  2.0  should  be 
delivered  to  a  Training  Support  Battalion  (TSBn)  that  supports  an  ARNG  unit  in  Tennessee  or 
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South  Carolina  that  uses  the  Mobile  CCTT.  During  this  stage  the  CITT  Web  Site  would  be 
hosted  at  Fort  Knox  to  help  facilitate  further  testing  and  refinement. 

The  CITT  Team  would  implement  near-term  implementation  and  fielding  by  delivering 
the  new  software  and  system  to  existing  CITT  sites  and  to  the  TSBn.  The  team  would  provide 
initial  user  training  for  commanders,  other  unit  trainers  and  CCTT  Site  staff.  The  team  would 
also  maintain  the  CITT  “Help  Desk”  and  conduct  evaluation  of  CITT  2.0  software  and  the  CITT 
Web  Site. 

Mid-Term 

The  mid-term  stage  provides  for  the  fielding  of  CITT  2.1  and  a  refined  Web  Site.  CITT 
2.1  software  would  be  fielded  to  existing  CITT  locations;  PCs  and  2.1  software  would  be  fielded 
to  the  newest  CCTT  sites  and  selected  units  supported  by  those  sites.  At  this  stage,  CITT  should 
be  fielded  to  each  brigade  supported  by  a  CCTT  Site.  Additionally,  each  TSBn  supporting  units 
using  Mobile  CCTTs  should  receive  a  laptop  with  CITT  2. 1  installed. 

Implementation  of  this  stage  becomes  a  joint  effort  between  the  development  team, 
proponent  schools,  and  PM-CATT.  The  CITT  Team  would  deliver  software  to  existing  sites  and 
provide  initial  user  training  as  in  the  near-term  stage.  This  would  be  done  as  part  of  the 
anticipated  follow-on  effort  to  CITT-2.  Once  this  initial  fielding  is  completed,  responsibility  for 
CITT  would  be  passed  to  either  PM-CATT  or  ATSC.  The  selected  agency  or  its  contractors 
would  complete  the  fielding  of  CITT  to  new  CCTT  Sites,  TSBn’s,  and  brigades  supported  by 
CCTT.  Additionally,  they  would  host  the  CITT  Web  Site  and  maintain  the  CITT  “Help  Desk.” 
The  final  part  of  this  stage  is  the  inclusion  of  CITT  initial  training  in  officer  basic  and  career 
courses  and  ANCOC  at  the  Armor  and  Infantry  Schools.  This  would  enable  units  to  take  full 
advantage  of  the  exercise  development  functions  and  training  information  that  are  a  part  of 
CITT.  It  also  provides  for  CITT  support  to  all  units,  schools  and  agencies  supported  by  CCTT 
and  sets  the  foundation  for  further  development  and  fielding  of  CITT.  Finally,  it  moves  CITT 
from  an  R&D  effort  to  a  production  effort  and  provides  for  the  sustainment  of  its  content  and 
functions. 

Long-Term 

Long-term  fielding  and  implementation  puts  CITT  (or  a  CITT-like  system)  into  the 
mainstream  of  Army  systems  and  provides  for  foil  fielding  of  CITT.  In  this  stage  the  objectives 
are  to  make  full  use  of  technology  to  support  Army-wide  distribution  of  data,  implement  the 
“To-Be”  CITT  design  and  field  a  system  that  supports  training  development  and  management  for 
unit  commanders  and  trainers.  This  includes  fielding  a  CITT-like  system  to  active  component 
battalions/squadrons,  Reserve  Component  TSBn’s  that  support  Enhanced  Brigades,  institutions, 
and  CCTT  Sites. 

In  this  stage  CITT  should  become  a  part  of  the  appropriate  ATIMP  system  and  link  to 
other  systems  as  designed.  The  CITT  servers  should  be  located  at  each  installation  that  has  a 
CCTT  site  to  facilitate  data  distribution.  Implementation  should  be  continued  as  in  the  mid-term 
stage,  however  updates/upgrades  would  be  provided  via  the  Web  whenever  possible. 
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This  stage  would  require  more  system  development  and  use  of  new  technology  to 
enhance  the  distribution  of  data.  The  CITT  would  become  akin  to  AS  AT  but  reach  into  units 
throughout  the  Army  and  enable  the  sharing  of  exercise  TSPs,  distribution  of  new  information  on 
training,  and  automatic  software  upgrades  via  the  Web. 

Table  2  shows  the  personnel  required  to  maintain  and  field  CITT  during  the  mid-  and 
long-term  implementation  and  fielding  stages.  This  staffing  is  required  to  provide  the  requisite 
upgrades,  updates  and  refresher  training  for  the  CITT.  The  personnel  listed  would  be  located  at  a 
central  site,  and  selected  members  would  be  tasked  to  create  a  Mobile  Training  Team  (MTT). 
This  MTT  would  travel  as  required  to  new  sites,  units,  and  schools  to  implement  upgrades, 
updates,  and  refresher  training. 

Table  2 

Staffing  Requirements  for  Mid-  and  Long-Term  Implementation  Strategy  and  Fielding  Plans 


Position 

Function 

Personnel 

MTT 

System 

Administrator 

In  addition  to  primary  duties:  responsible  for  the  operation  and 
maintenance  of  the  CITT;  maintains  baseline  hardware  and  software 
configurations;  conducts  preventative  and  corrective  maintenance  on  the 
CITT;  as  required  identifies  new  technologies  to  be  incorporated  into  the 
CITT. 

1 

X 

System 

Engineer 

In  addition  to  primary  duties:  Responsible  for  the  maintenance  and 
operation  of  the  “Help  Desk,”  troubleshoots  all  reported  faults  and  errors; 
develops  detailed  test  plans  for  diagnosing  CITT  hardware  and  software 
faults;  identifies  errors  and  their  causes;  reports  findings  to  CITT  users. 

1 

X 

Programmer 

In  addition  to  primary  duties:  Responsible  for  all  programming 
requirements  of  the  objective  CITT  and  supporting  web-based  application; 
designs  and  develops  enhancements  to  the  CITT  in  accordance  with 
guidance  received  from  the  Training  Developer  and  Military  Subject 

Matter  Expert. 

2 

Training 

Developer 

In  addition  to  primary  duties:  Responsible  for  all  training  development 
requirements  needed  to  support  the  CITT;  is  a  subject  matter  expert  in  the 
training  development  process  identified  in  US.  Army  Training  and 

Doctrine  Command  Regulation  350-70  (Department  of  the  Army,  1999); 
interfaces  with  CITT  users  to  ensure  full  understanding  of  the  embedded 
training  development  methodology;  interfaces  with  proponent  agencies  to 
ensure  full  compliance  with  evolving  doctrine;  as  required,  identifies  new 
features  to  be  incorporated  into  the  CITT. 

1 

Military 

Subject  Matter 
Expert 

In  addition  to  primary  duties:  Serves  as  resident  expert  on  military 
operations  for  mechanized  forces  (armor  and  infantry)  from  the  platoon  to 
brigade  level;  interfaces  with  CITT  users  and  proponent  agencies  to 
ensure  full  compliance  with  existing  military  operations  and  training 
doctrine;  as  required,  identifies  new  features  to  be  incorporated  into  the 
CITT. 

1 

Total 

6 

NA 
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Lessons  Learned 


There  are  a  number  of  lessons  learned  from  this  project  that  will  be  of  value  to  similar 
future  projects  as  well  as  to  the  fielding  and  implementation  of  the  CITT  itself. 

1.  The  CITT  provides  a  valuable  tool  to  the  user  which  can  aid  development  and  delivery 
of  training  designed  specifically  to  meet  the  unit’s  needs.  However,  it  is  not  a  “cookbook”  tool- 
it  does  not  replace  or  make  up  for  the  fact  that  the  user  needs  to  have  a  good,  working 
understanding  of  the  concepts,  principles,  and  practices  of  structured  training  development  and 
delivery.  The  most  successful  users  of  CITT  have  been  those  who  had  the  most  knowledge 
regarding  structured  training.  While  the  CITT  2.0  prototype  and  the  CITT  Web  Site  both  contain 
a  substantial  amount  of  information  on  structured  training,  it  would,  nevertheless,  be  of  great 
benefit  to  include  this  information  in  current  institutional  training.  It  appears  from  data  collected 
in  this  project  that  knowing  this  information  in  advance  greatly  facilitates  CITT  use  and  the 
products  produced. 

2.  Acceptance  of  the  CITT  will  be  difficult  if  units  perceive  it  as  “more  work.” 
Participants  in  both  the  current  project  and  the  previous  CITT  project  have  commented  on  the 
fact  that,  at  first  glance,  CITT  appears  to  take  a  good  deal  of  time  and  involve  a  good  deal  of 
effort.  Several  made  comments  such  as  “commanders  won’t  have  the  time  to  do  this.”  For  CITT 
to  succeed,  it  is  important  that  users  see  its  benefits  to  them  from  the  very  beginning.  It  must 
also  offer  the  user  a  process  that  is  as  efficient  as  possible  while  maintaining  the  effectiveness  of 
the  exercises  developed  using  it. 

3.  The  use  of  the  focus  group  in  the  current  project  was  much  more  efficient  than  in  the 
original  CITT  project.  In  the  original  project,  meetings  typically  lasted  an  hour  or  less  versus  the 
three  to  four  hour  sessions  in  this  project.  This  provided  sufficient  time  for  the  team  to  present 
information  at  a  meaningful  level,  and  for  the  focus  group  to  provide  considered  responses. 

4.  Even  though  there  was  a  high  level  of  cooperation  and  coordination  between  the  CEIT 
and  CITT  development  teams,  it  is  difficult  to  integrate  two  projects  that  have  different  timelines 
and  different  priorities.  In  CITT,  for  example,  the  development  of  the  shared  database  is  critical 
to  being  able  to  automatically  populate  appropriate  fields  in  the  TSP.  In  CEIT,  on  the  other 
hand,  the  shared  database  is  secondary  to  developing  the  reader  file  used  to  transfer  data  between 
CEIT  and  the  CCTT. 

5.  When  conducting  FE,  there  is  a  trade-off  between  maintaining  control  in  the  test 
environment  and  using  participants  at  their  “home”  location  who  are  doing  “real”  work.  At  Fort 
Hood,  for  example,  the  needs  of  the  unit  to  develop  real  exercises  within  a  short  time  frame 
superceded  the  needs  of  the  CITT  team  to  collect  FE  data  resulting  in  limitations  to  the  data 
collected  and  on  their  interpretation.  Early  in  a  system’s  development,  what  is  needed  is  to  test 
all  parts  of  the  system  in  a  predictable  way  so  that  one  can  see  how  the  various  components 
interact  and  what  needs  changed  or  improved.  This  can  be  accomplished  more  efficiently  using 
participants  who  are  representative  of  the  potential  user  population  and  who  are  provided 
specific  test  cases  or  scenarios. 
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6.  The  various  CCTT  sites  have  different  site-specific  procedures  for  developing  and 
modifying  exercises.  For  example,  the  time  required  between  submitting  an  exercise  and  having 
that  exercise  ready  to  run  in  CCTT  varies.  In  addition,  some  sites  involve  the  unit  in  proofing 
new  or  modified  exercises;  some  do  not.  In  order  for  CITT  to  be  used  to  its  maximum  potential, 
it  will  be  advantageous  to  have  a  common  set  of  exercise  management  procedures  across  all 
sites. 


7.  Related  to  item  6,  the  CCTT  sites’  primary  function  is  to  train  units,  not  to  write  TSPs. 
When  units  request  modifications  to  exercises,  the  sites  are  able  to  make  changes  to  the 
execution  files  readily;  however,  the  exercise  TSP  that  corresponds  to  the  exercise  is  not 
changed.  The  result  is  an  increasing  number  of  exercises  in  the  sites’  databases  of  executable 
exercises  that  have  no  corresponding  TSP  for  units  to  use  to  prepare  for  and  run  the  exercise. 

The  CITT  provides  the  most  appropriate  tool  for  the  sites  and/or  the  units  to  update  the  exercise 
TSPs;  however,  until  CITT  is  institutionalized  into  the  Army’s  training  system  for  the  CCTT,  it 
is  unlikely  that  this  updating  will  occur. 

8.  It  is  very  resource-intensive  to  develop  an  Internet  site  for  both  IE5  and  Netscape 
Navigator.  While  such  development  can  and  is  being  done  in  the  public  sector  where  both  IE5 
and  Navigator  are  used  extensively,  it  would  be  beneficial  to  the  Army  to  decide  on  one  browser 
for  use  across  the  Army.  This  would  eliminate  the  need  to  do  parallel  development. 

Summary  and  Recommendations 

This  report  has  described  the  R&D  efforts  to  expand  and  enhance  the  CITT  for  the 
CCTT.  The  CITT  will  provide  commanders  and  other  unit  trainers  with  the  capability  to  develop 
structured  training  exercises  for  use  in  the  CCTT  virtual  trainer  including  the  ability  to  select 
existing  training  exercises  that  match  their  unit’s  needs,  and  if  no  such  exercises  exist,  to  modify 
existing  exercises  or  create  new  ones.  The  primary  outcomes  of  the  project  include:  the  design 
for  the  enhanced  CITT  which  is  fully  documented  in  a  separate  report;  the  CITT  2.0  which  has 
been  refined  and  enhanced  based  upon  FE  results  as  well  as  upon  expanding  the  capabilities  of 
earlier  versions  to  include  exercises  for  digitally  equipped  units,  alternate  terrain  maps,  enhanced 
CEIT  features,  and  the  capability  to  upload  exercises  to  a  central  repository;  the  CITT  Web  Site 
which  provides  remote  access  via  the  Internet  to  a  repository  of  existing  CCTT  exercises  which 
users  can  review  and  download  if  desired,  extensive  information  on  CCTT  training  capabilities, 
structured  training  principles,  etc.,  a  newsgroup,  links  to  other  relevant  web  sites,  and 
information  on  each  of  the  CCTT  sites;  and  near-,  mid-,  and  long-term  implementation  and 
fielding  plans  for  the  CITT. 

It  is  recommended  that  future  follow-on  efforts  to  this  project  examine  or  consider  the 
following: 

1.  The  integration  of  FBCB2  into  CCTT  training  and  into  exercise  TSPs.  Where  should 
FBCB2  considerations  be  included;  how  should  they  appear  at  the  AAR  workstation;  and  how 
and  to  whom  should  the  AAR  workstation  operator  assign  FBCB2? 

2.  The  inclusion  of  formalized  training  in  implementation  and  fielding  of  CITT  at  both 
units  and  CCTT  sites.  This  training  should  emphasize  the  structured  training  process. 
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Additionally,  greater  emphasis  on  structured  training  should  be  incorporated  into  the  curriculum 
of  existing  Army  training  efforts  such  as  ANCOC  and  ACCC. 

3.  The  use  of  a  common  or  similar  GUI  for  both  the  CITT  and  the  Web  Site.  This  will 
make  it  easier  for  users  to  use  both. 

4.  The  implementation  and  support  of  the  CITT  Web  Site  in  the  long  term.  Even  after 
the  site  is  fully  developed,  it  is  highly  likely  that  it  will  require  a  high  level  of  maintenance  to 
keep  it  current  with  the  latest  Internet  technology  as  well  as  advances  in  the  virtual  training 
environment.  CITT  will  also  require  maintenance  to  keep  it  current  with  content  changes,  (e.g., 
tasks,  tactics,  techniques  and  procedures;  doctrine)  that  impact  training  in  the  CCTT. 

5.  The  plan  for  all  CCTT  sites  to  use  common  software  releases  beginning  January  2000. 
This  should  simplify  future  development  efforts. 

6.  The  use  of  both  test  cases  and  naturalistic  observation  in  future  FE  of  the  CITT.  Test 
cases  provide  more  control  of  the  specific  CITT  functions  tested,  while  naturalistic  observation 
of  users  in  their  home  environment  better  assesses  how  the  CITT  will  actually  be  used. 

7.  Exercise  management  including  the  approval  process  for  including  user-developed 
exercises  in  an  exercise  repository  available  to  other  Army  personnel. 
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Appendix  A 
Acronyms 


AA  assembly  area 

AAR  after  action  review 

ACCC  Armor  Captains  Career  Course 

AFRU  Armored  Forces  Research  Unit 

ANCOC  Advanced  Non-Commissioned  Officers  Course 

ARI  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

ARNG  Army  National  Guard 

ASAT  Automated  Systems  Approach  to  Training 

ATIMP  Army  Training  Information  Management  Program 

ATSC  Army  Training  Support  Center 

CATS  Combined  Arms  Training  Strategies 

CATT  Combined  Arms  Tactical  Trainer 

CCTT  Close  Combat  Tactical  Trainer 

CEIT  CCTT  Exercise  Initialization  Tool 

CITT  Commanders’  Integrated  Training  Tool 

CLS  contractor  logistics  support 

COR  Contracting  Officer’s  Representative 

DA  Department  of  the  Army 

EDUCCATT  Education  of  CCTT  through  computer  assisted  training  technology 

FBCB2  Force  XXI  Battle  Command  Brigade  and  Below 

FE  formative  evaluation 

EM  field  manual 

GUI  graphical  user  interface 

ID  identification 

IDEF  Integrated  Definition 

IE5  Internet  Explorer  5 

IO  Instructional  Overview 

LAN  Local  Area  Network 

MAJ  Major 

METL  mission  essential  task  list 

MTP  Mission  Training  Plan 

MTT  Mobile  Training  Team 

O/C  observer/controller 

OPFOR  opposing  forces 

A-l 


OPORD 

operations  order 

PI 

P2 

P3 

PM-CATT 

POC 

PVD 

Primary  1  (Temperate  Forest  Terrain  Database) 

Primary  2  (National  Training  Center  Terrain  Database) 
Primary  2  (Fort  Hood  Terrain  Database) 

Program  Manager  for  the  Combined  Arms  Tactical  Trainer 
point  of  contact 
plan  view  display 

R&D 

research  and  development 

SAF 

SATS 

SFC 

SOW 

STRICOM 

semi-automated  forces 

Standard  Army  Training  System 

Sergeant  First  Class 
statement  of  work 

U.S.  Army  Simulation,  Training,  and  Information  Command 

TDB 

TED 

TEXMIS 

TRADOC 

TREDS 

TSBn 

TSP 

terrain  database 
training  event  diagram 

Training  Module  Executive  Management  Information  System 
U.S.  Army  Training  and  Doctrine  Command 
tactical  reconnaissance  and  exploitation  demonstration  system 
training  support  battalion 

Training  Support  Package 

VB 

Visual  Basic 
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Usability  Assessment  Observation  and  Interview/Debriefing  Forms 
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DIRECTIONS  FOR  USE: 


Procedure 

Place  your  name  on  the  front  of  the  booklet.  Each  FE  observer  will  be  provided  with  his/her  individual 
copy. 

At  the  beginning  of  each  observation  session,  complete  the  Test  Case  Identification  information.  Record 
the  Participant’s  name,  the  date,  and  the  start  time  of  the  test. 

During  the  observation  session,  record  your  observations  on  the  portion  of  the  sheet  labeled 
“Observations.”  Each  time  the  participant  accesses  “Help”  put  a  tally  in  the  “Help”  column  at  the  point  in 
your  observation  at  which  Help  was  accessed.  The  Location  column  information  is  recorded  using  the 
Observation  Guidance  Form  and  indicates  where  the  Participant  was  in  CITT  when  the  observation  was 
made.  At  the  completion  of  the  session,  record  the  Stop  Time.  Use  as  many  recording  sheets  as  required 
for  the  session. 

Rationale 

In  accordance  with  the  Formative  Evaluation  plan,  we  are  particularly  interested  in  the  time  taken  to 
complete  each  module,  the  paths  the  user  takes  through  CITT,  whether  a  successful  outcome  was  reached, 
the  types  of  navigation  problems  that  occurred,  any  system  errors  that  occurred,  and  difficulties  the  user 
experienced  as  he/she  used  the  CITT.  (Other  FE  data  will  be  collected  through  surveys  and  interviews.) 
The  goal  is  to  capture  as  much  of  the  session  as  practical;  however,  based  on  prior  user  testing,  we  need  to 
recognize  that  we  can  not  record  every  action  taken  by  the  participant  nor  every  question/comment  made. 

Providing  Assistance 

At  the  bottom  of  the  Observation  Guidance  form  is  a  box  labeled  Exercise  Assistance.  This  shows  the 
progression  of  assistance  the  participant  should  be  provided  in  the  event  he/she  experiences  difficulties 
with  the  assessment.  Our  overall  goal  is  passive  observation;  however,  based  on  prior  user  testing,  it  is 
anticipated  that  assistance  will  occasionally  be  required. 

When  required,  observers  should  provide  the  minimum  assistance  necessary.  The  first  action  should  be  to 
suggest  that  the  participant  try  the  CITT  Help.  This  will  often  be  sufficient  to  solve  the  problem.  If  that 
is  not  successful,  provide  a  general  hint  or  assistance.  For  example,  “You  might  want  to  consider  the  task 
steps  you  want  the  unit  to  perform.”  If  that  is  still  not  sufficient,  provide  a  direct  instruction.  For 
example,  “Go  to  Select  Initial  Settings  and  list  the  task  steps  you  want  the  unit  to  perform.” 

As  assistance  is  provided  during  a  session,  it  is  important  that  the  details  be  captured  and  recorded.  We 
need  to  note  when  assistance  was  required,  the  nature  of  the  assistance  provided,  and  the  extent  of  the 
assistance  provided.  The  need  for  assistance  may  represent  a  flaw  or  weakness  in  CITT;  however,  we 
will  not  be  able  to  determine  this  unless  we  have  detailed  information  on  the  assistance  provided. 
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P: 

Sheet 

of 

Date: 

Start  Time: 

Stop  Time: 

Location: 

Observations: 

Help: 

Guidelines  for  Interview/Debriefing  Session  following  CITT  Usability  Assessment 

The  debriefing  should  occur  after  the  participant  has  completed  the  on-line  CITT  survey. 

The  debriefing  supplements  the  data  collected  during  observation  and  from  the  survey  and  can 
be  used  to  elaborate  on  data  previously  collected.  The  debriefing  should  be  conducted  using  a 
conversational  format;  however,  the  general  categories  listed  below  should  be  covered. 

1.  Follow  up  on  any  items/concems  remaining  from  the  CITT  assessment  observations,  for 
example,  any  questions  you  made  note  of  during  the  assessment  re:  why  a  participant  took  a 
particular  action. 

2.  Ask  participant  for  his  overall  impression  of  CITT  in  terms  of: 

•  Look  and  feel 

•  Ease  of  navigation 

•  Ease  of  use 

•  Map/Overlay  Tool 

3.  Ask  participant  about  the  information  included  in  Learn  About  CITT: 

•  Value  of  the  content 

•  Readability 

•  Use  of  audio  and  video 

•  Ease  of  navigation  (links,  navigation  bar,  etc.) 

4.  Ask  participant  about  the  on-line  help. 

•  Relevance  to  reason  he  accessed  help 

•  Satisfactorily  answered  his  questions 

5.  Ask  participant  about  Education  of  CCTT  through  computer  assisted  training 
technologies  (EDUCCATT). 

•  Did  he/she  access  EDUCCATT 

•  Did  EDUCCATT  provide  the  information/training  needed 

•  Overall  impression  of  EDUCCATT 

6.  Ask  participant  what,  if  anything  in  CITT,  made  it  difficult  to  complete  his  task. 

7.  Ask  participant  what  features  of  CITT  were  especially  useful. 

8.  Ask  participant  what,  if  anything,  he  would  like  to  have  had  in  CITT  that  wasn’t  included. 
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9.  Ask  participant  for  any  other  comments  he  wishes  to  provide. 
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Appendix  C 

Exercise  Execution  Observation  and  Interview/Debriefing  Forms 
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DIRECTIONS  FOR  USE: 


Procedure 

This  booklet  contains  the  recording  forms  for  use  in  the  Exercise  Execution  phase  of  the  CITT  1.1 
formative  evaluation  at  Fort  Hood.  Place  your  name  on  the  front  of  the  booklet. 

The  booklet  contains  two  types  of  recording  forms:  forms  for  Exercise  Execution  observations,  and 
interview/debriefing  forms  for  CCTT  Site  and  Unit  personnel. 

Exercise  Execution  Observations 

During  the  Exercise  Execution  activities,  record  your  observations  on  the  forms  provided.  In 
accordance  with  the  Formative  Evaluation  plan,  we  are  interested  in  whether  the  unit  was  able  to 
successfully  execute  the  exercises  selected,  modified,  or  created  using  CITT.  It  is  not  feasible  or 
necessary  to  provide  a  contemporaneous  record  of  the  Exercise  Execution  process;  rather,  we  are 
interested  in  actions  and  outcomes  which  indicate  difficulties  or  errors  in  the  execution  of  the 
exercises.  The  recording  forms  provide  space  for  the  following:  a  description  of  the 
error/difficulty,  and  your  analysis  of  the  factors  (CITT  and  non-CITT)  which  contributed  to  the 
difficulty/error. 

Contractor  Logistics  Support  (CLS)  and  Unit  Personnel  Interview/Debrief 

The  CLS  and  Unit  Personnel  Interview/Debrief  recording  forms  contain  questions  to  be  used 
during  the  debriefing  of  the  site  and  unit  personnel  who  were  involved  in  the  Exercise  Execution 
activities.  The  booklet  contains  separate  sections  for  the  site  and  unit  personnel.  The 
interview/debriefing  will  probably  require  30-60  minutes  and  should  be  conducted  as  a  group. 
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Exercise  Name:. 


Difficulty/Error: 


Contributing  factors 


Difficulty/Error: 


Contributing  factors 


Guidelines  for  CCTT  Site  Personnel  Debriefing  Session  Following  Exercise  Execution 


The  debriefing  should  occur  after  the  Exercise  Execution  of  all  exercises  has  been  completed. 

The  debriefing  supplements  the  data  collected  during  the  Exercise  Execution  observation.  The 
debriefing  should  be  conducted  as  a  group  activity  using  a  conversational  format;  however,  the 
items  listed  below  should  be  covered.  Note:  a  given  item  may  be  more  relevant  to  one  group  of 
participants  than  to  others. 

1 .  Follow  up  each  difficulty/error  recorded  from  the  Exercise  Execution  observations. 
Determine  what  the  Site  personnel  thought  went  wrong  and  whether  the  problem  can  be  related 
to  CITT  (i.e.,  would  the  difficulty/error  have  occurred  in  any  event  or  did  it  occur  as  a  result  of 
CITT  having  been  used  to  develop  the  exercise  and/or  train  the  unit  personnel)? 

2.  Cover  the  following: 

•  Have  any  of  the  site  personnel  used  the  CITT?  If  Yes,  what  did  they  use  the  CITT  for? 

•  As  related  to  building/proofing  the  exercises: 

1.  Did  the  exercise  materials  (Plan  Sheets,  Exercise  TSP)  include  sufficient  data  on  the  intent  of 
the  exercise? 

2.  Did  the  exercise  materials  (Plan  Sheets,  Exercise  TSP)  delivered  by  the  unit  include 
sufficient  data  to  build  the  exercise  initialization  file? 

3.  What  specific  data/information  was  needed  that  was  not  included  in  the  exercise  materials? 

•  For  those  site  personnel  involved  in  running/supporting/evaluating  the  exercise  (e.g.,  semi- 
automated  forces  [SAF]  operators,  AAR  Team),  did  the  Exercise  TSP  materials  provide 
sufficient  information  to  support  your  role  in  the  exercise?  If  not,  what  information  was 
missing? 
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Guidelines  for  Unit  Personnel  Debriefing  Session  Following  Exercise  Execution 

The  debriefing  should  occur  after  the  Exercise  Execution  of  all  exercises  has  been  completed  (if 
possible). 

The  debriefing  supplements  the  data  collected  during  the  Exercise  Execution  observation.  The 
debriefing  should  be  conducted  as  a  group  activity  using  a  conversational  format;  however,  the 
items  listed  below  should  be  covered.  Note:  a  given  item  may  be  more  relevant  to  one  group  of 
participants  than  to  others. 

1 .  Follow  up  each  difficulty/error  recorded  from  the  Exercise  Execution  observations. 
Determine  what  the  unit  personnel  thought  went  wrong  and  whether  the  problem  can  be  related 
to  CITT  (i.e.,  would  the  difficulty/error  have  occurred  in  any  event  or  did  it  occur  as  a  result  of 
CUT  having  been  used  to  develop  the  exercise  and/or  train  the  unit  personnel)? 

2.  Cover  the  following: 

•  Did  any  of  the  unit  personnel  use  the  CUT?  If  Yes,  what  did  they  use  the  CUT  for? 

•  For  each  Unit  Support  Workstation  Operator,  did  they  use  the  CUT  to  prepare  to  execute  the 
exercise?  If  so,  how  (EDUCCATT,  IO,  etc.)?  Did  the  preparation  they  received  in  CUT 
adequately  prepare  them  to  support  the  exercises?  If  not,  ask  for  specific  examples  of  lack  of 
preparation  or  lack  of  required  knowledge? 

•  Were  any  unit  personnel  involved  in  building/proofing  the  exercises  at  the  CCTT  site  (for 
those  exercises  developed  using  the  CUT)?  If  Yes,  did  the  CUT  produced  materials  provide 
sufficient  data/information  to  build/proof  the  exercise? 

•  (If  they  used  CUT)  What,  if  any,  features/functions  of  the  CUT  TSP  development  process 
were  particularly  helpful  in  the  execution  of  the  exercises? 

•  (If  they  used  CUT)  What,  if  any,  features/functions  of  the  CUT  TSP  development  process 
created  particular  difficulties  in  the  execution  of  the  exercises? 

3.  For  any  unit  personnel  who  used  CUT  to  develop  exercises: 

•  Did  the  exercises  as  executed  adequately  match  the  exercises  as  “intended”  when  they  were 
developed  in  CUT?  If  not,  ask  for  specific  instances  or  examples  of  a  mismatch.  How  are 
the  mismatches  related  to  the  CUT  (e.g.,  CUT  provided  incorrect  information,  insufficient 
information,  etc?  Lack  of  knowledge  of  the  cognitive  process  for  developing  structured 
training  exercises)? 

•  What,  if  any,  features/functions  of  the  CUT  TSP  development  process  were  particularly 
helpful  in  the  development  of  the  exercises? 

•  What,  if  any,  features/functions  of  the  CITT  TSP  development  process  created  problems  in 
the  development  of  the  exercises? 

•  What  would  they  most  like  to  see  added  to  CUT? 

•  What  could  be  removed  from  CUT? 
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Appendix  D 

Formative  Evaluation  Findings  from  Fort  Benning  and  ANCOC 


(To  facilitate  observation,  observers  used  an  aid  depicting  all  CFTT  activities  and  a 
corresponding  abbreviation.  In  the  following  listing,  the  abbreviations  indicate  the  specific 
activity  in  CITT  in  which  a  user  was  engaged,  e.g.,  CE1  indicates  “Create-Exercise  Outline- 
Exercise  Tasks  and  Steps.”) 

1 .  The  users  at  Fort  Benning  wanted  to  have  access  to  tasks  not  included  in  the  CITT. 
Specifically,  they  are  graded  on  some  non-mission  essential  task  list  (METL)  tasks  and/or  tasks 
not  included  in  EUM.  They  wanted  to  be  able  to  include  those  tasks  in  their  exercise. 

2.  The  CEIT  tools  were  not  present  and  could  not  be  loaded  under  the  login  the  participant  was 
using.  As  project  director  reported:  1.  The  unit  had  put  the  machine  on  the  network  and  had 
established  two  domains.  They  have  renamed  our  administrator  account.  So,  when  participant 
went  into  PowerPoint  under  his  user  account  (which  does  have  administrator  privileges)  no 
tactical  reconnaissance  and  exploitation  demonstration  systems  (TREDS)  (CEIT)  tool  bars 
showed  up.  We  tried  to  install  the  tool  bars  with  our  instructions  and  got  an  error  that  said 
“PowerPoint  found  an  error  that  it  cannot  correct,  you  should  save  your  presentation  and  restart 
PowerPoint.”  We  did  this  and  it  still  did  not  work...  This  is  when  observers  called  developers. 
The  unit  administrator  came  and  told  observer  what  he  had  done  to  the  admin  account.  Tool  bars 
loaded  but  did  not  work  when  we  logged  on  with  the  new  admin  account.  Only  the  add-in  tools 
developer  wrote  worked.  We  got  a  Run  Time  70  error.  2.  At  this  point  developer  had  observer 
change  the  domain  from  an  NT  domain  to  a  Workgroup  domain.  This  did  nothing;  tool  bars 
loaded  but  did  not  work.  3.  Observer  tried  to  unload  the  TREDS  tools  but  this  did  not  work 
either.  Finally,  observer  copied  the  CITT  database  and  tried  to  uninstall  CITT.  The  computer 
said  it  uninstalled  but  the  database  and  a  number  of  files  were  still  in  the  system.  Observer 
manually  deleted  CITT  but  could  not  reinstall  it;  ran  out  of  memory.  Observer  defragged  the 
hard  drive  and  installed  CITT.  Only  the  CITTSA  CD  was  reinstalled  here.  This  time  the  tool 
bars  were  there  and  worked  but  only  for  the  renamed  ADMINISTRATOR  account.  NOTE: 
uninstalling  and  reinstalling  PowerPoint  never  did  work! ! ! !  Observer  tried  this  earlier. 

3.  We  need  to  develop  Help  for  CEIT  if  the  team  developing  it  doesn’t.  In  the  present  CEIT,  if 
the  user  hits  FI,  he  gets  PowerPoint  help,  not  context  specific  CEIT  help. 

4.  It  would  be  helpful  for  the  user  if  the  CEIT  install  placed  the  new  toolbars  in  some  logical 
order  and  location.  For  example,  observer  arranged  them  on  the  Benning  machine  so  that  they 
were  all  arranged  on  one  side  of  the  screen  and  were  in  the  order  they  need  to  be  applied.  This 
would  help  avoid  some  of  the  Runtime  5  errors  the  users  encountered. 

5.  A  Jet  database  error  was  produced  in  the  Select  Mission  Tasks. 

6.  On  Create  Mission  Set  Parameters,  the  training  event  diagram  (TED)  should  not  appear. 

7.  Participant  was  never  able  to  get  the  “Display  Contents”  in  PowerPoint  to  work. 
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8.  Got  a  Runtime  error  when  trying  to  print  posterized.  This  happened  one  time.  The  next 
time,  the  print  posterized  worked. 

9.  After  partitioning  into  three  exercises,  in  CE1,  user  had  all  tasks  in  the  drop  down  list,  not 
just  those  previously  selected  in  CM. 

10.  After  partitioning,  the  user  went  into  CE  and  was  completing  the  exercise  outlines  for  all 
exercises  simultaneously.  (This  may  not  be  a  particularly  good  way  to  do  it,  but  it’s  what  he 
did.)  He  frequently  lost  track  of  which  exercise  he  was  currently  in.  We  had  to  ask  him  several 
times  if  he  was  aware  that  he  was  in  exercise  3.  Invariably,  he  wasn’t. 

1 1 .  After  completing  CE,  participant  thought  he  was  finished.  He  had  been  watching  the 
navigation  indicator  that  shows  up  on  the  left  side  of  the  screen  and  all  the  boxes  had  turned 
green.  He  thought  he  was  finished  with  create.  Thus  he  closed  Create.  He  had  to  go  into 
Modify  to  complete  the  TSPs. 

12.  Asa  direct  result  of  what  happened  in  1 1 ,  when  participant  got  to  Commo  Matrix  in  Modify, 
there  was  none.  Users  need  to  be  able  to  go  back  into  Create. 

13.  We  need  to  deliver  CITT  with  a  formalized  training  process. 

14.  Participant  requested  that  FM  101-5-1  be  included  in  the  available  FMs. 

15.  The  CITT  needs  to  have  access  to  the  IO  from  any  screen. 

16.  Navigation  issue:  There  is  still  confusion  in  navigating  between  the  use  of  tabs  and  the 
“next”  and  “previous”  buttons.  2.0  will  solve  this  hopefully. 

17.  It  would  be  helpful  if  Events  could  be  auto  re-numbered  when  a  user  adds  a  new  event. 

18.  At  CG3,  user  wanted  to  add  a  new  task  step.  He  couldn’t.  Had  to  go  back  to  CE1. 

19.  On  the  Event  Guide,  users  do  not  indicate  CISs.  May  go  away  with  the  revised  CEIT. 

20.  Users  had  problems  with  the  current  process  for  printing  the  TSPs.  The  CITT  needs  to  be 
able  to  print  TSPs  (or  components  thereof)  without  having  to  go  through  print  preview. 

21.  Print  from  the  File  menu  should  be  disabled. 

22.  Exercise  Outline  printed  landscape  the  first  time  user  tried  to  print  it.  When  we  tried  to 
correct  it,  we  got  a  Dr.  Watson  error. 

23.  In  CE1,  got  a  “help  topic  not  found  error.”  Need  to  check  all  help  links. 

24.  Participant  in  MT1  and  MT2  had  difficulty  keeping  track  of  which  task  he  was  in. 
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25.  In  the  tutorial  or  in  training,  need  to  clarify  that  modifying  an  exercise  does  not  destroy  or 
alter  the  exercise  being  modified.  We  had  to  convince  participant  that  when  he  selected  the 
exercise  he  had  previously  created  to  modify,  he  wasn’t  going  to  lose  his  previous  exercise. 

26.  Is  it  possible  to  do  spell  check  in  Access?  Also  to  do  Find  and  Replace? 

27.  In  PowerPoint,  at  one  point  participant  had  a  very  large  blank  screen  with  a  very  small  map. 

28.  Participant  liked  the  fact  that  he  could  toggle  the  map  on  and  off  in  PowerPoint.  He  did  so 
frequently  allowing  him  to  get  a  clear  view  of  the  overlay  itself. 

29.  There  are  a  number  of  points  in  the  TSP  development  process  where  the  user  needs  to  be 
able  to  print.  For  example,  after  selecting  tasks/task  steps,  user  should  be  able  to  print  the  list 
selected  for  later  use. 

30.  “Save  and  Return”  does  not  close  Word  or  PowerPoint. 

31.  The  Task  Steps  column  on  the  Event  Guide  is  too  narrow. 

32.  Eliminate  the  “Close”  button  on  the  individual  forms. 

33.  User  was  in  PowerPoint  working  with  the  TED  or  the  Overlay  (happened  both  ways); 
minimized  PowerPoint  so  he  could  work  in  CITT.  When  he  maximized  PowerPoint,  the  TED  or 
Overlay  was  gone. 

34.  User  was  in  Review  -  PreExercise.  Tried  to  make  a  change,  but  couldn’t.  So  he  hit 
“previous”  two  or  three  times,  navigating  backwards,  but  this  produced  an  error  message. 

35.  In  Create,  “Complete”  should  not  be  the  default.  Because  it  is,  if  the  user  does  not  complete 
his  exercise  in  Create  and  forgets  or  otherwise  does  not  uncheck  this  box,  when  he  goes  into 
modify,  it  makes  another  exercise. 

36.  Need  to  add  task  force  MTPs. 
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Appendix  E 

Formative  Evaluation  Findings  from  Fort  Hood  and  ACCC 

(To  facilitate  observation,  observers  used  an  aid  depicting  all  CITT  activities  and  a 
corresponding  abbreviation.  In  the  following  listing,  the  abbreviations  indicate  the  specific 
activity  in  CITT  in  which  a  user  was  engaged,  e.g.,  CE1  indicates  “Create-Exercise  Outline- 
Exercise  Tasks  and  Steps.”) 

1.  The  unit  had  put  the  CITT  machine  on  their  network.  When  observers  arrived,  CEIT  tools 
weren’t  working  even  for  the  Administrator.  Also,  they  could  not  be  installed.  When  they  took 
the  machine  off  the  network  and  tried  to  restart,  the  machine  died.  Turned  out  to  be  a  hard  drive 
failure. 

2.  On  the  laptop-the  unit  had  reinstalled  NT  and  Office;  however,  they  had  not  installed  the 
assistant.  When  CITT  attempted  to  load,  it  failed  because  it  couldn’t  locate  the  assistant.  We 
talked  about  this  last  week  and  developer  is  looking  at  code  to  bypass  assistant  and  agent  if  they 
are  not  loaded. 

3.  On  the  laptop,  the  video  quality  of  the  demos  is  poor.  The  screen  is  shaky/jumpy. 

4.  The  CITT  uninstall  does  not  work. 

5.  Got  Dr.  Watson  errors  on  Print  Matrix  OPORD.  This  was  on  the  machines  as  observers 
found  them.  So  probably  related  to  what  the  unit  had  done  to  the  machine.  After  reloading  NT, 
Office,  and  CITT,  didn’t  get  the  message  again. 

6.  In  CE1,  there  is  insufficient  space  in  the  field  to  show  the  entire  task  step  description.  Need 
to  increase  the  size  of  the  field. 

7.  In  CE2,  we  need  to  make  it  clear  to  the  user  that  he  can  crop  the  image  again  (i.e.,  He  has 
previously  cropped  the  image  when  he  created  the  overlay.).  Now,  when  he  gets  to  create  TED, 
he  can  crop  the  previously  cropped  image. 

8.  In  PowerPoint,  “regroup”  did  not  work  for  unit  symbols. 

9.  Got  a  Visual  Basic  (VB)  error  in  PowerPoint  when  the  user  tried  to  designate  an  OPFOR 
unit. 

10.  In  MC4,  user  couldn’t  replace  workstations.  Also,  we  don’t  have  intelligence  officer  and 
operations  and  training  officer  workstations  (radios). 

1 1 .  MX2  -  user  was  not  able  to  complete  the  activities. 

12.  Conflict  between  CITT/CEIT  capability  for  identifying  Unit  IDs  and  bumper  numbers  and 
CCTT’s  capability  to  use  these  IDs.  The  CCTT  unit  topology  editor  should  solve. 
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13.  In  ME,  sometimes  task  steps  did  not  appear. 

14.  In  ME,  user  selected  a  task  step  then  decided  he  didn’t  want  it.  He  could  not  delete  it. 

15.  In  MW2,  user  had  to  reenter  OPFOR  starting  locations. 

16.  Equipment  State  drop  downs  on  the  Plan  Sheet  do  not  function  properly.  This  may  be 
obsolete  in  2.0,  but  we  need  to  monitor  how  it  shows  up  in  CEIT. 

17.  When  printing  the  Plan  Sheet,  the  last  row  is  not  printed. 

18.  On  the  Plan  Sheet,  there  are  two  “Remarks”  columns. 

19.  User  suggested  screening  thumbnails  by  criteria  (i.e..  He  had  entered  select  criteria  and  had 
gotten  a  list  of  matching  exercises.).  When  he  went  to  look  at  the  thumbnails,  he  would  have 
preferred  if  “view  all  thumbnails”  only  cycled  through  the  matching  exercises,  not  the  whole  set 
ofTSPs. 

20.  There  was  a  problem  with  the  wheel  mouse.  Text  disappeared  for  no  apparent  reason. 

21 .  User  suggested  having  an  “Add  All”  button  for  task  steps. 

22.  In  selecting  task  steps,  user  wanted  the  drop  down  list  to  return  to  the  last  step  selected 
rather  than  to  the  beginning  of  the  list. 
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